Overview Use Cases Connectors Authentication Getting Started with the Python SDK API API Reference API Classes API Inputs AIAgentBatchPredictionArgs AIAgentTrainingConfig AbacusUsageMetricsDatasetConfig AccuracyBelowThresholdConditionConfig AgentConversationMessage AgentFlowButtonResponseSection AlertActionConfig AlertConditionConfig ApplicationConnectorDatasetConfig AttachmentParsingConfig Base64ImageResponseSection BatchPredictionArgs BiasViolationConditionConfig Blob BlobInput BoxDatasetConfig ChartResponseSection ChatLLMBatchPredictionArgs ChatLLMPredictionArguments ChatLLMTrainingConfig ClusteringTimeseriesTrainingConfig ClusteringTrainingConfig CodeResponseSection CollapseResponseSection ConfluenceDatasetConfig ConstraintConfig ConstraintProjectFeatureGroupConfig CrawlerTransformConfig CumulativeForecastingPredictionArguments CumulativeForecastingTrainingConfig CustomAlgorithmTrainingConfig CustomTrainedModelTrainingConfig DataGenerationConfig DataIntegrityViolationConditionConfig DatabaseConnectorExportConfig DataframeResponseSection DatasetConfig DatasetDocumentProcessingConfig DecisionNode DeployableAlgorithm DocumentClassificationTrainingConfig DocumentProcessingConfig DocumentRetrieverConfig DocumentSummarizationTrainingConfig DocumentVisualizationTrainingConfig EmailActionConfig EventAnomalyTrainingConfig ExtractDocumentDataConfig FeatureDriftConditionConfig FeatureGroupExportConfig FeatureMappingConfig FeatureStorePredictionArguments FieldDescriptor FileConnectorExportConfig ForecastingBatchPredictionArgs ForecastingMonitorConfig ForecastingPredictionArguments ForecastingTrainingConfig FreshserviceDatasetConfig GoogleAnalyticsDatasetConfig GoogleDriveDatasetConfig HistoryLengthDriftConditionConfig HotkeyPrompt ImageUrlResponseSection IncrementalDatabaseConnectorConfig ItemAttributesStdDevThreshold JSONSchema JiraDatasetConfig KafkaDatasetConfig LastNMergeConfig ListResponseSection MarkdownConfig MergeConfig MonitorFilteringConfig MonitorThresholdConfig NSamplingConfig NamedEntityExtractionBatchPredictionArgs NamedEntityExtractionTrainingConfig NaturalLanguageSearchPredictionArguments NaturalLanguageSearchTrainingConfig OneDriveDatasetConfig OperatorConfig OptimizationPredictionArguments OptimizationTrainingConfig OutputVariableMapping ParsingConfig PercentSamplingConfig PersonalizationBatchPredictionArgs PersonalizationTrainingConfig PredictionArguments PredictionCountConditionConfig PredictiveModelingBatchPredictionArgs PretrainedModelsBatchPredictionArgs ProjectFeatureGroupConfig ProjectFeatureGroupTypeMappingsConfig PythonFunctionArgument RegressionPredictionArguments RegressionTrainingConfig ResponseSection RestrictFeatureMappings ReviewModeProjectFeatureGroupConfig RuntimeSchemaResponseSection SamplingConfig Segment SentenceBoundaryDetectionBatchPredictionArgs SentenceBoundaryDetectionTrainingConfig SentimentDetectionTrainingConfig SftpDatasetConfig SharepointDatasetConfig StdDevThreshold StreamingConnectorDatasetConfig SystemConnectorTool TargetDriftConditionConfig TeamsScraperDatasetConfig TextResponseSection ThemeAnalysisBatchPredictionArgs ThemeAnalysisTrainingConfig TimeWindowConfig TimeWindowMergeConfig TimeseriesAnomalyPredictionArguments TimeseriesAnomalyTrainingConfig TrainablePlugAndPlayBatchPredictionArgs TrainingConfig TriggerConfig UnionTransformConfig UnpivotConfig VectorStoreConfig WorkflowGraph WorkflowGraphEdge WorkflowGraphNode WorkflowNodeInputMapping WorkflowNodeInputSchema WorkflowNodeOutputMapping WorkflowNodeOutputSchema WorkflowNodeTemplateConfig WorkflowNodeTemplateInput WorkflowNodeTemplateOutput ZendeskDatasetConfig Documentation Chat Bot API Search How to

CrawlerTransformConfig

Transform a input column of urls to html text

KEY	TYPE	Description
userAgent	str	If provided, will use this user agent instead of randomly selecting one.
depthColumn	str	Increasing depth explores more links, capturing more content
crawlDepth	None	None
inputColumnType	None	None
disableHostRestriction	bool	If True, will not restrict crawling to the same host.
honourWebsiteRules	bool	If True, will respect robots.txt rules.
outputColumn	str	Name of output column to store transformed data.
inputColumn	str	Name of input column to transform.