Custom Python Transformation (UDF) | Knowledge Base (Product User Guides)

Custom Python Transformation (UDF) is an Action Node for Standard Data Flows that lets advanced users write custom Python logic when prebuilt nodes are not enough. Use it to transform records row by row or operate on an entire dataset using a DataFrame.

Contact support if this Action Node is not enabled for your account.

At a glance
UDF supports Python 3.11, offers row-based and DataFrame-based processing modes, includes editor validation and preview capabilities, and supports 200+ approved libraries.

What this Action Node does

Custom Python Transformation (UDF) allows you to create your own transformation logic directly in a Data Flow. It is intended for use cases that cannot be handled cleanly with standard configuration-only nodes.

Row-based mode: best for simple per-record logic such as cleaning, combining, or deriving fields.
DataFrame-based mode: best for larger datasets, multi-column transformations, filtering, aggregations, and vectorized operations.
Failure controls: optionally continue processing when some rows fail, or stop based on a configured failure threshold.
Outputs: transformed file output, job logs, and, in applicable cases, an error file with failure reasons.

Setting	Description
Mode	Processing mode is determined by the structure of the Python code, not by a separate setting. Use the `transform_row(row)` function for row-based processing, or work with `input_df` and `output_df` for DataFrame-based processing.
Python code	Provide the transformation logic to run in Python 3.11.
Failure threshold	Optionally fail the step if the percentage of failed rows exceeds a specified limit.
Continue on failure	Optionally skip failed rows and continue processing remaining records.

Input Requirements and File Behavior

UDF does not require a minimum number of columns. A single-column file is supported.

There is no minimum column-count requirement for UDF inputs.

Supported input types for UDF include:

CSV
XLSX
JSON
JSONL

Headers and column mapping behavior depend on the file type and the metadata available to the Data Flow. In practice, the most important requirement is that the platform must correctly identify the input file format before the UDF step runs.

Common issue: if the platform cannot detect the input format, the file may be labeled as unknown. In that case, the UDF step fails before your Python code runs.

This explains why a two-column CSV may appear to work while a one-column CSV fails. The problem is not the number of columns. The problem is usually file type detection.

Unsupported file type: unknown

Recommended Configuration

When the input format may be ambiguous, add a File Type Action Node before the UDF step.

This is especially recommended for:

Single-column files
Files with unusual or ambiguous delimiters
Fixed-width style inputs
Sample uploads where auto-detection may be inconsistent

To configure it:

1. Open the Data Flow in the builder.

2. Between the inflow and the UDF step, add the File Type Action Node.

3. Set File Type to csv for CSV input, or to the appropriate type such as jsonl.

4. Save the Data Flow.

5. Run the Data Flow again.

Recommended default: if you are working with a single-column CSV and the UDF fails unexpectedly, set File Type explicitly to csv before troubleshooting the Python code.

Optional Configuration

By default, your downstream behavior may depend on the rest of the Data Flow configuration. If downstream systems require a specific output format, add an Output File Format or related output-formatting Action Node after the UDF step.

This is useful when:

A downstream step expects JSONL instead of CSV
You need specific header behavior or delimiters
You want to standardize output formatting after a custom transformation

Examples

DataFrame-based example

Use DataFrame mode when you need table-level context or better performance on larger files.

Python

import hashlib
output_df['email_md5'] = input_df['email'].apply(lambda x: hashlib.md5(x.encode()).hexdigest())
output_df.drop(columns=['email'], inplace=True)

Row-based example

Use row-based mode when you need lightweight logic for each individual record.

Python

import hashlib
def transform_row(row):
    row['email_md5'] = hashlib.md5(row['email'].encode()).hexdigest()
    return row

Best Practices

Choose DataFrame mode for large datasets, multi-column logic, filtering, or aggregation.
Choose row-based mode for straightforward record-level transformations.
Start simple and validate with sample input before applying more complex logic.
Check the file type first if the UDF appears not to run at all.
Use failure controls intentionally so partial data loss does not go unnoticed.

Troubleshooting

Symptom	Likely cause	Resolution
`Unsupported file type: unknown`	The platform could not detect the input format	Add a File Type step before the UDF and set it to `CSV` or the correct format.
Works with 2 columns, fails with 1	Detection relies on delimiters or file characteristics, not a UDF column minimum	Add a File Type step or otherwise make the file format explicit.
UDF code never runs	The failure happens before the Python runtime executes the script	Check file type metadata and upstream file-format settings before debugging code.
Unexpected downstream format	Output format was not explicitly configured	Add an output-format Action Node after the UDF step.