Dataset Webhook Export

Export datasets asynchronously and receive the results via webhook when processing is complete. This is ideal for large datasets that would timeout with synchronous requests, or when you want to process data in the background without maintaining an active connection.

How It Works

1. Submit Export Request: Send your dataset query along with a webhook URL
2. Receive Confirmation: Get immediate 201 response with export queued
3. Processing: Your export is processed asynchronously in the background
4. Webhook Notification: Your endpoint receives the results when complete

Request Format

The request requires:

• dataset: Your query configuration (columns, filters, limits)
• webhookURL: HTTPS endpoint where results will be sent
• name: Descriptive name for your export
• format: Either ‘csv’ or ‘jsonl’

Webhook Response

When your export completes, you’ll receive a POST request with this payload:

1
{
2
  "queryUUID": "123e4567-e89b-12d3-a456-426614174000",
3
  "name": "Seattle Rental Data",
4
  "format": "csv", 
5
  "payload": { /* your original dataset query */ },
6
  "gcsBlob": {
7
    "url": "https://storage.googleapis.com/...",
8
    "expires": "2024-02-15T10:30:00Z",
9
    "timeTakenMs": 45000,
10
    "numberRows": 12847
11
  },
12
  "requestedOn": "2024-01-15T10:00:00Z",
13
  "completedOn": "2024-01-15T10:00:45Z"
14
}

$
     curl --location 'https://api.hellodata.ai/dataset/export' \
>
--header 'Content-Type: application/json' \
>
--header 'x-api-key: your-api-key' \
>
--data '{
>
    "dataset": {
>
        "scopes": [
>
            {"column": "street_address"},
>
            {"column": "asking_rent", "aggregate": "Avg"}
>
        ],
>
        "filters": [
>
            {"column": "msa", "filter": {"equals": "Seattle, WA"}},
>
            {"column": "bed", "filter": {"in": [1, 2, 3]}}
>
        ],
>
        "limit": 10000
>
    },
>
    "webhookURL": "https://yourapp.com/webhook/dataset-export",
>
    "name": "Seattle Rental Data Export",
>
    "format": "csv"
>
}'

Implementing Your Webhook Handler

Your webhook endpoint should handle the incoming POST request and verify the HMAC signature for security. The signature is included in the SHA256-HMAC-Signature header. You’ll use the body of the request and your API key to create a SHA256 HMAC signature. This signature can be compared to the one in the header in order to verify that the request has come from us.

1
const express = require("express");
2
const crypto = require("crypto");
3
const app = express();
4
5
app.use(express.json());
6
7
app.post("/webhook/dataset-export", (req, res) => {
8
  // Get the HMAC signature from the header
9
  const signature = req.get("SHA256-HMAC-Signature");
10
  if (!signature) {
11
    return res.status(401).send("Invalid signature");
12
  }
13
14
  // Verify webhook authenticity
15
  const hmac = crypto.createHmac("sha256", "your-api-key");
16
  hmac.update(JSON.stringify(req.body));
17
  const expectedSignature = hmac.digest("hex");
18
19
  if (
20
    !crypto.timingSafeEqual(
21
      Buffer.from(expectedSignature, "hex"),
22
      Buffer.from(signature, "hex"),
23
    )
24
  ) {
25
    return res.status(401).send("Invalid signature");
26
  }
27
28
  const { queryUUID, gcsBlob } = req.body;
29
30
  // Download and process your data
31
  // processExportedData(gcsBlob.url);
32
33
  console.log(`Export ${queryUUID} completed:`);
34
  console.log(`- Download URL: ${gcsBlob.url}`);
35
  console.log(`- Rows: ${gcsBlob.numberRows}`);
36
  console.log(`- Processing time: ${gcsBlob.timeTakenMs}ms`);
37
38
  res.status(200).send("OK");
39
});
40
41
app.listen(3000, () => { console.log("Server listening on port 3000"); });

Security & Verification

Always verify the HMAC signature to ensure the webhook request is legitimate:

1. Compute HMAC: Use your API key and the payload
2. Compare Signatures: Match against the SHA256-HMAC-Signature header
3. Reject Invalid Requests: Return 401 for signature mismatches

Important Notes

• Processing Time: Large exports may take several minutes to complete
• File Format: Files are compressed with gzip for efficient transfer
• Webhook Requirements: Your endpoint must be HTTPS and publicly accessible
• Response Expected: Your webhook should respond with 2xx status to confirm receipt
• Tracking: Use the queryUUID to match requests with responses

Error Handling

If the export fails during processing, your webhook will not be called. Monitor your webhook endpoint for delivery failures.

Use Cases

Data Pipeline Integration

1
// Automatically trigger analysis when new data arrives
2
app.post('/webhook/dataset-export', (req, res) => {
3
    const { gcsBlob, name } = req.body;
4
    
5
    // Download the data
6
    const csvData = await downloadFile(gcsBlob.url);
7
    
8
    // Trigger your data pipeline
9
    await triggerDataPipeline(csvData, name);
10
    
11
    res.status(200).send('OK');
12
});

Scheduled Reports

1
# Process weekly market reports
2
@app.route('/webhook/dataset-export', methods=['POST'])
3
def process_weekly_report():
4
    data = request.get_json()
5
    
6
    if 'weekly' in data['name'].lower():
7
        # Send to reporting system
8
        send_to_reporting_dashboard(data['gcsBlob']['url'])
9
    
10
    return 'OK', 200

Full Documentation

Find complete technical details in our API Reference.