BigQuery JavaScript UDF

Question

Bu sorular muhtemelen

BigQuery UDF memory exceeded error on multiple rows but works fine on single row

başka örneğidir Ama ben bir soru yerine cevap olarak yazılan önerildi "Aşıldı Kaynakları" ile başarısız olur.

Logfiles dosyasını bir tabloya ayrıştırmak için javascript kullanıyorum. Javascript ayrıştırma işlevi nispeten basit. 1M satırlarında çalışır, ancak 3M satırlarında başarısız olur. Log dosyaları 3M'den çok daha büyük olabilir, bu yüzden bir problemdir.

Bu işlev aşağıdadır.

function parseLogRow(row, emit) { 

    r = (row.logrow ? row.logrow : "") + (row.l2 ? " " + row.l2 : "") + (row.l3 ? " " + row.l3 : "") 
    ts = null 
    category = null 
    user = null 
    message = null 
    db = null 
    seconds = null 
    found = false 
    if (r) { 
     m = r.match(/^(\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d\.\d\d\d (\+|\-)\d\d\d\d) \[([^|]*)\|([^|]*)\|([^\]]*)\] ::(\(([\d\.]+)s\))? (.*)/) 
     if(m){ 
      ts = new Date(m[1])*1 
      category = m[3] || null 
      user = m[4] || null 
      db = m[5] || null 
      seconds = m[7] || null 
      message = m[8] || null 
      found = true 
     } 
     else { 
      message = r 
      found = false 
     } 
    } 

    emit({ 
     ts: ts, 
     category: category, 
     user: user, 
     db: db, 
     seconds: seconds*1.0, 
     message: message, 
     found: found 
     }); 
    } 


    bigquery.defineFunction(
    'parseLogRow',       // Name of the function exported to SQL 
    ['logrow',"l2","l3"],     // Names of input columns 
    [ 
     {'name': 'ts', 'type': 'float'}, // Output schema 
     {'name': 'category', 'type': 'string'}, 
     {'name': 'user', 'type': 'string'}, 
     {'name': 'db', 'type': 'string'}, 
     {'name': 'seconds', 'type': 'float'}, 
     {'name': 'message', 'type': 'string'}, 
     {'name': 'found', 'type': 'boolean'}, 
    ], 
    parseLogRow       // Reference to JavaScript UDF 
); 

Bu sorgu ile işleve başvuran ediyorum:

SELECT 
    ROW_NUMBER() OVER() as row_num, 
    ts,category,user, 
    db,seconds,message,found, 
FROM parseLogRow((SELECT * FROM[#{dataset}.today] 
     LIMIT 1000000 
    )) 

Bazı örnek verileri 'bugün' tablosundaki (CSV olarak) şöyle:

logrow,l2,l3 
# Logfile created on 2015-12-29 00:00:09 -0800 by logger.rb/v1.2.7,, 
2015-12-29 00:00:09.262 -0800 [INFO|7aaa0|] :: Running scheduled job: confirm running gulp process,, 
2015-12-29 00:00:09.277 -0800 [DEBUG|7aaa0|] :: Restarted gulp process,, 
2015-12-29 00:00:09.278 -0800 [INFO|7aaa0|] :: Completed scheduled job: confirm running gulp process,, 
2015-12-29 00:00:14.343 -0800 [DEBUG|7aaa2|scheduler] :: Polling for pending tasks (master: true),, 
2015-12-29 00:00:19.396 -0800 [INFO|7aaa4|] :: Running scheduled job: confirm running gulp process,, 
2015-12-29 00:00:19.409 -0800 [DEBUG|7aaa4|] :: Restarted gulp process,, 
2015-12-29 00:00:19.410 -0800 [INFO|7aaa4|] :: Completed scheduled job: confirm running gulp process,, 
2015-12-29 00:00:29.487 -0800 [INFO|7aaa6|] :: Running scheduled job: confirm running gulp process,, 
2015-12-29 00:00:29.500 -0800 [DEBUG|7aaa6|] :: Restarted gulp process,, 
2015-12-29 00:00:29.500 -0800 [INFO|7aaa6|] :: Completed scheduled job: confirm running gulp process,, 
2015-12-29 00:00:39.597 -0800 [INFO|7aaa8|] :: Running scheduled job: confirm running gulp process,, 
2015-12-29 00:00:39.610 -0800 [DEBUG|7aaa8|] :: Restarted gulp process,, 
2015-12-29 00:00:39.611 -0800 [INFO|7aaa8|] :: Completed scheduled job: confirm running gulp process,, 
2015-12-29 00:00:44.659 -0800 [DEBUG|7aaaa|scheduler] :: Polling for pending tasks (master: true),, 
2015-12-29 00:00:49.687 -0800 [INFO|7aaac|] :: Running scheduled job: confirm running gulp process,, 
2015-12-29 00:00:49.689 -0800 [DEBUG|7aaac|] :: Restarted gulp process,, 
2015-12-29 00:00:49.689 -0800 [INFO|7aaac|] :: Completed scheduled job: confirm running gulp process,, 
2015-12-29 00:00:59.869 -0800 [INFO|7aaae|] :: Running scheduled job: confirm running gulp process,, 
2015-12-29 00:00:59.871 -0800 [DEBUG|7aaae|] :: Restarted gulp process,, 

Bu bir hack bir bit gibi ben bir 3 sütun tablosu (gerçekten, sadece bir sütun) günlük olarak bir sınırsız sekme ile bir CSV olarak içe aktarma olarak içe aktarırken (biz genellikle günlük dosyasında herhangi bir sekme yok), tabloya dönüştürmek için bir sorgu kullanarak actu müttefik.

Bu deseni seviyorum çünkü ayrıştırma hızlı ve dağıtılmış (çalışırken).

Başarısız olan bir iş: bigquery-looker: bquijob_260be029_153dd96cfdb. Tekrarlanabilir bir vakaya ihtiyacınız varsa lütfen bana ulaşın.

Herhangi bir yardım veya öneri takdir edilecektir.

Answer 1

Nokta 1.
Ben UDF ile sorunu görmüyorum - kaldırmak ve çalışması gerekir - bu Sorunun doğrusu() AŞIRI ROW_NUMBER() kullanımında olduğunu düşünüyorum 10M sıralar
bile benim için çalıştı!

SELECT 
    ts,category,user, 
    db,seconds,message,found, 
FROM parseLogRow((SELECT * FROM[#{dataset}.today] 
)) 

Point 2.
prospektif performans bakınca aşağıdaki hızlı çalışmalıdır (Sanırım) ve "düz" BQ

SELECT 
    REGEXP_EXTRACT(logrow, r'^(\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d\.\d\d\d (?:\+|\-)\d\d\d\d) \[[^|]*\|[^|]*\|[^\]]*\] :: .*') AS ts, 
    REGEXP_EXTRACT(logrow, r'^(?:\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d\.\d\d\d (?:\+|\-)\d\d\d\d) \[([^|]*)\|(?:[^|]*)\|(?:[^\]]*)\] :: (?:.*)') AS category, 
    REGEXP_EXTRACT(logrow, r'^(?:\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d\.\d\d\d (?:\+|\-)\d\d\d\d) \[(?:[^|]*)\|([^|]*)\|(?:[^\]]*)\] :: (?:.*)') AS user, 
    REGEXP_EXTRACT(logrow, r'^(?:\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d\.\d\d\d (?:\+|\-)\d\d\d\d) \[(?:[^|]*)\|(?:[^|]*)\|([^\]]*)\] :: (?:.*)') AS db, 
    REGEXP_EXTRACT(logrow, r'^(?:\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d\.\d\d\d (?:\+|\-)\d\d\d\d) \[(?:[^|]*)\|(?:[^|]*)\|(?:[^\]]*)\] :: (.*)') AS message, 
    REGEXP_MATCH(logrow, r'^((?:\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d\.\d\d\d (?:\+|\-)\d\d\d\d) \[(?:[^|]*)\|(?:[^|]*)\|(?:[^\]]*)\] :: (?:.*))') AS found 
FROM (
    SELECT logrow +IFNULL(' ' + l2, '') + IFNULL(' ' + l3, '') AS logrow 
    FROM YourTable  
)