# Welcome WorkflowAI is an open-source platform where [product](/getting-started/for-product-managers) and [engineering teams](/getting-started/for-developers) collaborate to build and iterate on [AI features](/ai-features-playbook/what-is-an-ai-feature). If you're new to WorkflowAI, we recommend you to read the [playbook](/ai-features-playbook/introduction) to learn more about how to build, test, improve, and optimize your AI features. You can also read about the different features WorkflowAI offers to make using AI easier, different concepts within WorkflowAI, how to get set up and build AI features with [WorkflowAI Cloud](/workflowai-cloud/introduction), and how to use our [Python SDK](/python-sdk/get-started) for a developer-focused experience. We're glad you're here; have a look around! This documentation should contain answers to most of your questions. If you can't find what you're looking for here, we encourage you to ask a question in the [Q\&A section of our GitHub Discussions](https://github.com/WorkflowAI/WorkflowAI/discussions/categories/q-a) or [on Discord](https://discord.gg/auuf8DREZh) # For Product Managers AI features can help you bring intelligence to your product, using the same AI models powering ChatGPT.\ WorkflowAI is designed to empower product managers to build and optimize AI features, **without writing code**. > "I’ve always had great ideas for AI features, but engineering resources were always the bottleneck. Now, with WorkflowAI, I’ve been able to build and improve AI features myself—no coding, no waiting.\ > I can quickly go to market, gather customer feedback, and iterate fast." > > \~ Perri Gould, Head of Product at BerryStreet ## Why Product Managers like WorkflowAI: ### Go from idea to AI feature. No code needed. Stop waiting on bandwidth. WorkflowAI lets anyone on your team design, test, and deploy AI features using plain language - no code needed. ### Tired of black-box AI? Now you can see inside. When the AI gets it wrong, you need to know why — fast. WorkflowAI logs every input and output automatically, so you can spot issues, understand what went wrong, and fix it without waiting on engineering. ### No more switching tabs and spreadsheets to compare models. WorkflowAI shows you outputs, costs, and latency side-by-side, all in one view. ### Edit prompts. Skip the tickets. Tired of creating tickets just to tweak a prompt? WorkflowAI lets you update prompts instantly, no engineering work required. Go from feedback to fix in seconds. ## How to get startedL Read our [AI Features Playbook](/ai-features-playbook/introduction) to learn how to build your first AI feature in minutes. {% hint style="info" %} If you need support while building your first AI feature, you can reach out to us on [Github](https://github.com/WorkflowAI/WorkflowAI/discussions) or [Discord](https://discord.gg/auuf8DREZh). {% endhint %} # For Developers > “Being provider-agnostic used to mean maintaining multiple complex integrations. With WorkflowAI, we can seamlessly switch between LLM providers without any extra integration effort or overhead—saving us engineering time and headaches.” > > \~ Aymeric Beaumet, CTO at M1 ## Why Software Engineers like WorkflowAI: ### Our [Python SDK](/python-sdk/get-started) Stop wasting time maintaining separate integrations for every LLM. WorkflowAI gives you unified, seamless access to all models through a single, clean API. ### Structured outputs WorkflowAI ensures your AI responses always match your defined structure, simplifying integrations, reducing parsing errors, and making your data reliable and ready for use. ### Write code only when you want to WorkflowAI gives you flexibility: quickly prototype new AI features via our intuitive web interface, or dive directly into code whenever you need deeper customization and control. ### Proudly [open-source](https://github.com/WorkflowAI/workflowai) WorkflowAI is fully open-source with flexible deployment options. Run it self-hosted on your own infrastructure for maximum data control, or use the managed [WorkflowAI Cloud](/workflowai-cloud/introduction) service for hassle-free updates and automatic scaling. ### Model-agnostic Works with all major AI models including OpenAI, Anthropic, Claude, Google/Gemini, Mistral, DeepSeek, Grok with a unified interface that makes switching between providers seamless. ### [Streaming supported](https://github.com/WorkflowAI/documentation/blob/main/docs/features/code/README.md#streaming) Enables real-time streaming of AI responses for low latency applications, with immediate validation of partial outputs ## How to get started: Read our [AI Features Playbook](/ai-features-playbook/introduction) to learn how to build your first AI feature in minutes in our web-app, or learn more about our [Python SDK](/python-sdk/get-started) to build features programmatically. # Creating and Managing Organizations WorkflowAI allows you to create organizations so you can easily collaborate on building features with your whole team. ## How do I create an organization? 1. Log into WorkflowAI 2. Select your name at the bottom of the sidebar to open the user menu. If you are not a member of any organization: you will see a button to create your first organization. ![What you will see if you're not part of an org](https://2418444523-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FW4ng0K5LfFjYqHYuPgNh%2Fuploads%2Fgit-blob-4b97773b166ca900b65e7cc7d8bf7cb4080f6213%2Fcreate-an-org.png?alt=media) If you are already a member of an organization: go to [workflowai.com/organization/select](https://workflowai.com/organization/select) and select **Create Organization** from there. ## How can I join an existing organization? If the organization you want to join has Automatic Invitations enabled, and your email matches the criteria: 1. Sign up for WorkflowAI 2. You should be automatically added to the organization. However if you do not see the organization appear when you log in: go to [workflowai.com/organization/select](https://workflowai.com/organization/select) and select the organization you want to join from the list. ![Organization selector](https://2418444523-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FW4ng0K5LfFjYqHYuPgNh%2Fuploads%2Fgit-blob-7c70acab8ba6f77521ad55ccee0f0411e75d1482%2Fselect-an-org.png?alt=media) If the organization you want to join does not have Automatic Invitations enabled or your email does not match the criteria to be automatically admitted, you will need to ask an existing member of the organization to send you an invitation. {% hint style="info" %} If you are the admin of an organization and want to learn how to enable Automatic Invitations, please see [Organization Joining Settings](#organization-joining-settings). {% endhint %} ## Can I be part of more than one organization? Yes! ## How can I switch between organizations? We are working on adding a better UX, but in the meantime you can switch between organizations by: 1. Logging into WorkflowAI 2. Going to [workflowai.com/organization/select](https://workflowai.com/organization/select) 3. Selecting the organization you want to view. ## Organization Joining Settings ### Verified Email Domains Verified email domains can be used to streamline enrollment into an organization. If your organization has a verified email domain, users can join the organization automatiaclly or by suggestion (see [Invitation Settings](#invitation-settings)) by signing up with an email address that matches the verified domain. Note: A verified domain cannot be a disposable domain or common email provider. For example, you cannot create a verified domain for @gmail.com. #### Enabling Verified Email Domains 1. Log into WorkflowAI 2. Select your name at the bottom of the sidebar to open the user menu. 3. Tap **Organization Settings** 4. In the **Verified Domains** section select and select **Add Domain** and enter the domain you want to verify. ![](https://2418444523-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FW4ng0K5LfFjYqHYuPgNh%2Fuploads%2Fgit-blob-66b1d5a9c45305414afc188b9403da4f9b73200c%2Fverify-domain.png?alt=media) ### Invitation Settings There are three different settings that determine how users can join an organization. In order to change the settings from the default of **No automatic enrollment**, you will need to first enabled Verified Domains. 1. **Automatic invitations**: Users are automatically invited to join the organization when they sign-up and can join anytime. 2. **Automatic suggestions**: Users receive a suggestion to request to join, but must be approved by an admin before they are able to join the organization. 3. **No automatic enrollment (default)**: Users can only be invited manually to the organization. ![Invitation settings](https://2418444523-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FW4ng0K5LfFjYqHYuPgNh%2Fuploads%2Fgit-blob-3b01388fd5e5ae1cd011eea6b0bbc7e815d4e8bb%2Finvitation-settings.png?alt=media) # Introduction {% hint style="success" %} This playbook was designed for both product and engineering teams. {% endhint %} ## Why should you read this playbook? Do you want to use AI to make your product even better? Whether to unlock feature functionality only possible with AI, or simply save time and money by optimizes processes, that's where we come in. At WorkflowAI, we make it easy to build AI features that can help your business in just a few minutes. Developing great AI features is a process of continous improvement, setting up evaluations, and collaboration between product and developments teams, but this playbook aims to help you every step of the way. ### With this playbook, you will learn: * What AI features are and what they can do * How to define an AI feature in WorkflowAI * How to test your AI feature * How to evaluate your AI feature * How to add your AI feture into your product * How to improve your AI feature # What is an AI Feature? AI Features are mini-programs that use AI algorithms (LLMs) as their brain to accomplish tasks typically provided by users or other AI features. The AI feature understands the task requirements, plans a sequence of actions to achieve the task, executes the actions, and determines whether the task has been successfully completed. Some examples of what AI features can do: * **Summarize** long pieces of text * **Browse** a company URL and extract the list of their customers * **Search** the web to find the answer to a question * **Generate** a product description from an image * **Extract** structured data from a PDF, image, and other file types * **Classify** the sentiment of a customer message * **Scrape** a website and extract structured data For more inspiration on AI features you can build, sign up and log in at [WorkflowAI.com](https://workflowai.com) and select the **+ New** button to see a wide variety of example features. ## What is *not* an AI Feature? An AI feature should involve a single input-to-output interaction. Combining multiple sequences of inputs and outputs would instead constitute a workflow, which is not currently supported. In the event that there is a task that is better suited for a workflow, break the process into multiple agents that each handle one portion of the task only. * **Valid AI Feature:** "Extract calendar events detected in a thread of emails." * **Invalid AI Feature (ie. a Workflow):** "Extract calendar events from a thread of emails and then automatically send invitations for the events to guests." # Defining your AI Feature {% hint style="success" %} For product and engineering teams {% endhint %} ## Articulating the Goal of your AI Feature When thinking about building your own AI feature, it’s important to consider three key aspects: 1. **Goal:** Clearly describe how input data transforms into output data, ex. extracting calendar events detected in a list of provided emails. 2. **Input:** The type of data the feature will start with, ex. a list of emails. 3. **Output:** The type of data the feature will produce, ex. a list of calendar events. ## Creating your Schema {% hint style="info" %} We recommend building your first features directly in the [web-app](https://workflowai.com/) as opposed to the [Python SDK](/python-sdk/get-started), as the web app is the fastest way to get started. {% endhint %} Once you’ve established a clear goal for your AI feature, tap **+ New** on the web-app and write a few sentences describing what the agent should do. Based on your description, the AI feature builder will identify specific fields required for the input and output. Collectively, these fields create your schema. Some schemas are straightforward. For example: summarizing an article, typically involves one input field (the article) and one output field (the summary). Other features may need multiple fields. For instance, an input composed of a thread of emails might have separate fields for email content, senders, recipients, and timestamps. Initially, include all obvious fields, but don't worry about perfection. Schemas can be adjusted later as necessary. For guidance on addressing schema issues, refer to the section all about [improving your AI Feature](/ai-features-playbook/improving-your-ai-feature). Once you're happy with your first schema, tap **Save and Try in Playground**. From there, WorkflowAI will automatically generate a first set of instructions for the feature and a first input to use for testing. Congratulations, your AI agent is running! You can move on to testing and improving it on the playground. # Testing your AI Feature {% hint style="success" %} For product and engineering teams {% endhint %} ## Determine Feature Feasibility The first thing to determine after a first schema of the feature has been created is whether the goal of the feature is possible to accomplish with existing LLM limitations. After the first run of the new feature on the playground: confirm whether the feature resembles what was originally described and - if not - what is wrong with it? Can the issues be overcome using the techniques described in [Improving your AI Feature](/ai-features-playbook/improving-your-ai-feature)? Or are the issues indicative that the feature is beyond the limitations of current LLMs? Sometimes this answer isn’t immediately evident and becomes clearer over time. But getting an initial feel is important. #### A note about diagnosing issues with your AI Feature Addressing issues with your AI feature are an inherent part of building a successful feature and can occur at any stage. Problems generally happen more frequently early in the development process and generally decrease over time as you optimize your feature to align more closely with your goal. ## Test with Various Inputs After creating your schema, the first thing to do is test the feature by running it with varied inputs to ensure reliability across multiple scenarios. Aim for 10-20 diverse inputs depending on feature complexity. Options for generating test data include: * **Early Internal/Beta Deployment (recommended):** Deploy an early version internally or to a beta group and gather real-world data using WorkflowAI [observability and feedback features](/features/user-feedback). * **Generated Data:** WorkflowAI can automatically generate suitable test inputs. It’s possible to provide specific instructions to guide the generation process. * **Imported Input:** Input can be directly imported on the playground to facilitate testing. ## Test with Various Models WorkflowAI provides access to 70 different LLM models that can be used for running features. Different models will have different strengths - some are known for being highly intelligent, others are very cheap, and still others will excel in performing tasks quickly. Ultimately a prompt only needs to work for one model, as in the end, only one version will be chosen to be deployed at a time. When selecting the best model for a feature, consider the following capabilities: * **Intelligence:** Is the model able to perform the task requested and produce the correct output for the feature? More complicated features will require more advanced models. * **Price:** What is the estimated run volume of the feature and the associated budget? If there is a limited budget for a high volume feature, it might be necessary to pick a cheaper model. * **Latency:** How fast does the feature need to return an output? If a feature is user-facing, it’s generally best to avoid high latency models as the waiting time can lead to a poor user experience. Background features can generally get away with slower models as the pace is less noticeable. # Evaluating your AI Feature {% hint style="success" %} For product and engineering teams {% endhint %} When it comes evaluating the runs of AI feature, there are two types of features: * Features that are very easy to generate inputs for on the playground * Features that require more complicated, real-world inputs to evaluate Before spending time reviewing your runs, first determine if the feature can be accurately evaluated using runs with generated data from the playground, or if an early deployment to an internal beta is necessary in order to evaluate your feature using real data. ## Reviewing Runs Individual runs outputs can be evaluated on whether their output is correct or incorrect for a given input or not. This creates a quantitative baseline to determine feature accuracy on different versions. Runs can be reviewed in two places: * **Playground:** if the feature is easy to test using generated inputs, runs can be reviewed on the playground as they’re completed * **Runs Page:** if runs are coming from the API/SDK (or if you want to add a review from the playground after the fact), runs can be reviewed any time after they’re completed from the Runs page. There are two types of reviews: * **Human-Review:** a human reviewer manually evaluates a run and marks it as correct or incorrect. When a human review is left, you will see that the corresponding icon and background are solid. ![Human Review](https://2418444523-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FW4ng0K5LfFjYqHYuPgNh%2Fuploads%2Fgit-blob-474912f01bd6960dea36570a21a3c82f8b27affe%2Fhuman-review.png?alt=media) * **AI-Powered Review:** an AI agent evaluates a run and marks it as correct or incorrect. AI-powered reviews require an initial human review on the same input as a baseline for the AI review. When an AI review is left, you will see that the corresponding icon is highlighted, but not filled in. AI reviews can always be overriden by a human reviewer, so if something does not look right, you can correct the review. ![AI Review](https://2418444523-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FW4ng0K5LfFjYqHYuPgNh%2Fuploads%2Fgit-blob-983cc3bc3e26b77ef733185e51efe0c4ca71561d%2Fai-review.png?alt=media) {% hint style="info" %} **What's the point of AI-powered reviews?** AI-powered reviews save time by automatically reviewing runs with an input that matches an already human-reviewed run. So when you're iterating on your prompt with the same input, you won't need to manually review every single run. AI-powered reviews are also used to calculate accuracy scores for versions when benchmarking. [Learn more about how benchmarking works.](#benchmarking-versions) {% endhint %} ### Why is reviewing runs important? When you leave a review on a run, you're adding that input/output pair - and whether is it correct - to your Reviews dataset. This dataset is used to benchmark the performance of different versions of your feature. The more runs that are evaluated, the larger your Reviews dataset will be, which will create a more accurate result for your benchmarks. [Learn more about how benchmarking works.](#benchmarking-versions) Additionally, the more reviews that are added, the more the AI reviewer agent will understand all the criteria that determines correctness for your feature, and it will be able to handle evaluating runs on your behalf more effectively. ### Can more than one output be considered correct? In many cases, yes. For example, if there is a feature meant to summarize an article, there may be multiple summaries that contain all the correct information, but are phrased differently. In these cases, it's helpful to leave a review on all correct outputs. The multiple reviews help the AI reviewer agent get a better understanding of all the critieria that determines correctness. ### How big should my evaluation dataset be? How many runs should I review? In most cases, we recommend reviewing 10-20 separate inputs to build an initial dataset. However the actual number may vary depending on your use case. The more complicated your feature is, the more reviews should be added to ensure that you have a robust dataset that covers all your important use cases. ## Benchmarking Versions ![Benchmarks](https://2418444523-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FW4ng0K5LfFjYqHYuPgNh%2Fuploads%2Fgit-blob-3b9024db975cfe8e38019de701d2e82303c9153c%2Fbenchmark-table.png?alt=media) Benchmarking evaluates performance on a version level by comparing different versions' performance, helping to identify the most effective versions based on accuracy, cost, and latency. Benchmarks use the [reviews](#reviewing-runs) added about individual runs to calculate the overall accuracy of a version based on how many runs were viewed as correct vs. incorrect. ### How do I benchmark an AI Feature? In order to benchmark an AI Feature, you need to have two things: * Reviewed runs (we recommend starting with between 10-20 reviews, depending on the complexity of your AI Feature). You can learn more about how to review runs [here](/features/reviews). * At least two saved versions of the AI Feature on the same Schema. To save a version, locate a run on the Playground or Runs page and select the "Save" button. This will save the parameters (instructions, temperature, etc.) and model combination used for that run. After creating a review dataset and saving versions of your AI feature that you want to benchmark, access the Benchmark page in WorkflowAI's sidebar and select the versions you want to compare. The content of your review dataset will automatically be applied to all selected versions to ensure that they are all evaluated using the same criteria. {% embed url="" %} ### How exactly does benchmarking work? #### Accuracy Version accuracy is based off of the human reviews left on runs and supplimented by AI-powered reviews. The process goes as such: 1. AI feature runs are given reviews by a human reviewer. The review runs are added to the Reviews dataset page (visible on the [Reviews page](/features/reviews)). 2. When benchmarking a version, the selected version runs all the inputs present in the Reviews dataset. 3. Using the human reviews from the dataset as a baseline, the AI-powered reviews are added to evaluate any runs on the benchmarked version that don't yet have a human review. 4. The amount of correct and incorrect runs - based on both the human reviews and AI-powered reviews - are used to calculate the accuracy of the version. #### Price Price is calculated based on the number of tokens used in the version's runs and cost of the tokens for the selected model. #### Latency Latency is calculated based on the time it takes for each of the version's runs to complete. ### How can I add a specific input to my benchmark evaluation dataset? If there are inputs that you want to ensure are included when benchmarking a version, all you need to do is review at least one run of that input. Once an input has been reviewed, it will be automatically be added to your Reviews dataset and ultilized when benchmarking. ### A new model was released, how can I quickly evaluate it? We're actively working on an even faster way to evaluate new models, but in the meantime here is the current process we recommend: To quickly benchmark a new model: 1. Locate the feature you want to test the new model on 2. Make sure the schema selected in the header matches the schema you have deployed currently. 3. Go to the versions page and locate your currently deployed version (you will recognize it by the environment icon(s) next to the number) 4. Hover over the version and select **Clone** and then select the new model. 5. Go to the benchmark page and select both the currently deployed version and the new version you just created. Note: in order to ensure that a benchmark generates a fair and accurate comparison, it's important that you have a large enough evaluation dataset. See [How big should my evaluation dataset be/how many runs should I review?](#how-big-should-my-evaluation-dataset-be-how-many-runs-should-i-review) for more information. # Adding your AI Feature to your Product (and Beyond) After finding a suitable version of the feature, it can be integrated into your product by deploying it. ## Deploying a Version Once a suitable version is identified, the recommended next step is deploying the version to a WorkflowAI environment. WorkflowAI provides three environment options: dev, staging, production. A single version can be deployed to one or multiple environments. Deploying a version enables your product’s codebase to reference the version with an environment variable (any of the three mentioned above) rather than a hardcoded version number. Using environment variables simplifies the process of updating versions; updating a deployed version on WorkflowAI automatically updates the version the environment variable references within your product. No engineering intervention required! *Note: deploying a version on a different schema will require a code update. Schema updates are considered breaking changes, thus necessitating hardcoded version numbers in the product's codebase.* ## Integrating Your WorkflowAI Feature into Your Codebase {% hint style="success" %} For software engineers {% endhint %} After deploying your chosen version to an environment: 1. Go to the Code page. 2. Select your programming language. 3. Install the WorkflowAI package 4. Copy the provided integration code and paste into your codebase. Be sure to select the desired **version with the desired environment icon** from the version-selection dropdown to ensure the correct version is referenced in the generated code. ### Monitoring Runs Once deployed, it’s recommended to monitor real-time feature runs (via the [Runs](/concepts/runs) page) through WorkflowAI to help identify and rectify early issues. All Runs of a feature are logged automatically on the Run’s page, so they can be accessed and viewed any time. If issues with the feature are observed, refer to the [Improving your AI Feature](/ai-features-playbook/improving-your-ai-feature) section for common issues and tips on how to resolve them. ## Integrating WorkflowAI Feedback into Your Codebase {% hint style="success" %} For software engineers {% endhint %} After integrating the feature into your product, consider integrating WorkflowAI's [feedback component](/features/user-feedback) into your product to collect user insights directly. The feedback component allows you to easily collect user feedback on your WorkflowAI feature to inform ongoing feature improvement. ### Monitoring Feedback User feedback is essential for continuous improvement. Monitoring responses through WorkflowAI’s feedback system helps determine where enhancements are needed. The WorkflowAI feedback integration also connects to Slack so you'll be able to have user feedback sent from the WorkflowAI web app to Slack to make it easier to discuss the feedback insights with your team. ## Sharing your AI Feature ### If your AI feature is public If you want to share your public AI feature with others, you can do so by sharing the feature's URL with them. If your AI feature is not public but you would like it to be: 1. Login to [WorkflowAI](https://workflowai.com/) 2. Select **Settings** on the side bar of your feature 3. Select **Change AI Agent Visibility** and confirm ### If your AI feature is private If your AI feature is private, you can share the feature with others by inviting them to your organization. To invite others: 1. Login to [WorkflowAI](https://workflowai.com/) 2. Go to the Code page of your feature 3. Select **Invite Team** 4. Enter the email of the person you want to invite Once the user accepts the invitation, they will be added to your organization and will have access to the feature. # Improving your AI Feature Issues can occur at two levels: schema-level and version-level. The issue type determines the options available to make corrections.\ Below you will find some examples of common schema-level and version-level problems. ## Schema-Level Issues #### Problem: My feature is missing an input or output field When the schema fields don't align with expected inputs or outputs - such as missing fields - update the feature’s schema to resolve. The schema can be updated in the WorkflowAI web-app or via code directly. Examples of this problem: * A feature where a transcript of a discussion is extracting calendar events, but the input is missing the `transcript_time` parameter. * A feature where a transcript of a discussion is extracting calendar events, but the output is missing the `event_time` parameter. #### Problem: My feature has an unwanted field, or the name of a field is wrong When the schema fields don't align with expected inputs or outputs - extra fields, or incorrect field names - update the feature’s schema to resolve. The schema can be updated in the WorkflowAI web-app or via code directly. #### Problem: A field format is wrong (Schema-Level) Certain format issues require schema updates; others do not. * **Schema Update Required:** Issues involving changes in data structure, such as switching a field from a single string to a list of strings, require the schema to be updated * **No Schema Update Required:** Formatting adjustments (eg. specific format for dates, bullet points vs. paragraphs, or language requirements) can be resolved by updating field descriptions or examples within the playground. Refer to the [Version-Level Issues section](#problem-a-field-format-is-wrong-version-level) for more information on this case. ## Version-Level Issues Version-level issues have a wide variety of solutions depending on their nature. The first part of this section highlights common version-level problems and their solutions at a high level. The second part of this section provides further information about each solution. #### Problem: A field format is wrong (Version-Level) As noted in the schema-level issues section, certain format issues can be solved on the playground directly; others can not. * **Playground-Resolvable Cases:** Content format issues (dates, bullet points, language style) can be fixed directly in the playground by editing [instructions](#updating-instructions), or editing [examples, and descriptions](#updating-field-examples-and-descriptions). The edits can be completed with the help of the Playground Chat Agent or manually. * **Schema Update Required:** Data structure changes require updating the schema directly. Refer to the [Schema-Level Issues](#problem-a-field-format-is-wrong-schema-level) section for more information. #### Problem: My feature doesn’t seem to understand the output that I want Example: a task that should produce a summary in bullet points, but the summary is written as a paragraph. There are several solutions to try: * [Update the playground instructions](#updating-instructions) to ensure that the expected output behavior is clearly described. * Example of this problem that would be solved by this solution:A feature that should produce a one paragraph summary, but currently the summary is 2 or more paragraphs. Ensuring that the instructions specifically mention the one paragraph maximum will help ensure the LLM is aware of the size limit * [Updating a field’s examples and descriptions](#updating-field-examples-and-descriptions) can provided even clearer guidance for issues with a specific field, * Example of this problem that would be solved by this solution: A feature that should produce a summary in bullet points, but currently the summary is in a paragraph form. Updating the summary examples to include bullet points can help guide the LLM in formatting. * [Use highly intelligent models](#trying-different-models). For complicated prompts, or prompts where the instructions, examples, and descriptions have all been updated to no avail, experiment with different models, specifically models with higher intelligence. The playground chat agent can recommend high intelligence models, and model intelligence scores can also be viewed when hovering over models in the playground model dropdown. If even highly intelligent models are not able to produce the correct output, the prompt may be too complex for current LLM capabilities. See [Determine Feature Feasibility](/ai-features-playbook/testing-your-ai-feature#determine-feature-feasibility) for more information on feature limitations. #### Problem: The output does not contain up-to-date information LLMs have knowledge cut-offs and may produce outdated information. Integrating tools into instructions helps the LLM real-time data access. A list of available tools is available in the [Adding Hosted Tools to Instructions](#adding-hosted-tools-to-instructions) section. Example of this problem: * asking an LLM the current stock price of AAPL without including any @search tool. #### Problem: I’m getting a lot of errors when I use tools Tools help models perform different functionality that they would otherwise be unable to do. For example, accessing real-time data that is more recent than their knowledge cut-offs.\ Some models manage tools better than others; experimenting with [different models](#trying-different-models) will help you find the models that work best with the tools you need to use. #### Problem: I’m getting errors related to a model’s context window An LLM's context window refers to the maximum number of tokens (words or subwords) that the model can consider simultaneously when generating a response. In simpler terms, it's how much recent information the model can "remember" or use as context during a conversation or task. Models have varying context window sizes. If there is an error related to a feature exceeding a model’s context window, [switch models](#trying-different-models) to one that has a larger context window. The playground chat agent can provide recommendations for models with large context windows, and you can also view model’s context windows by hovering a model in the playground model dropdown to view its details ### Additional Guidance on Version-Level Solutions: #### Updating Instructions: Instructions provide overall guidance on how the agent should approach tasks, including what tools to use, what tone to adopt, and general behavioral guidelines. They're best for defining the agent's overall approach and methodology. Models will respond to the same instructions differently. When improving the instructions, remember that the instructions don’t need to work for every model on the playground. Ultimately, they only need to work for one model, as in the end, only one version will be chosen to be deployed at a time.\ Instructions can be updated with playground agent assistance or manually on the playground. #### Updating Field Examples and Descriptions: Examples and descriptions provide specific formatting guidance for individual fields. They show the agent exactly what the output should look like for each field and provide context about what each field should contain. Examples and Descriptions edits are made on the playground by: * **In all cases:** Asking the playground agent to make an update on your behalf by describing the change * **If output is currently displayed:** Hovering over a field name, then hover over the description/examples modal and tap Edit * **If no output is displayed:** Hovering over a description or example and tap Edit #### Updating Temperature: Temperature affects a model’s balance between precision and creativity: * **Lower (near 0):** More deterministic, consistent outputs that strictly follow instructions. * **Higher (near 1):** More creative, varied outputs with potentially more exploration The temperature can be adjusted on the playground in the temperature section underneath the instructions. #### Adding Hosted Tools to Instructions: Tools help models perform different functionality that they would otherwise be unable to do. Integrate WorkflowAI-hosted tools to provide real-time data from internet searches and scraping websites. Tools can be added by describing the use case to the playground chat agent or enabled manually using the buttons at the bottom of the instructions text field.\ Hosted tools that are currently available: * **@browser-text:** browses websites for information (text-only) * **@perplexity-sonar-pro:** browses the web for information using perplexity: a premier search offering with search grounding, supporting advanced queries and follow-ups. * **@search-google:** browses the web for information using google (not the default browser tool, should only be suggested as an alternative to perplexity, if perplexity is not performing as desired) #### Trying Different Models: Each LLM model offers distinct strengths and weaknesses. Experimenting with models directly in the playground or through agent recommendations helps identify the best model for your feature complexity and performance requirements. Different models can be selected using the model dropdowns on the playground. Model recommendations for a use case can be provided by the playground chat agent. # Image Generation (NEW!) Image generation features are simply features that have one or multiple images in the output.\ Image generation AI features are created in the same way as other AI features in WorkflowAI, either through our\ web app at [WorkflowAI.com](https://workflowai.com), or through our SDK. The models supporting image generation can be either specific to image generation like Imagen or GPT Image 1, or\ can handle both image an text outputs like Gemini Flash 2.0 Exp. ![Image Generation of a Sunset](https://2418444523-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FW4ng0K5LfFjYqHYuPgNh%2Fuploads%2Fgit-blob-8800db711da149401a87b65cffdb5f2099141163%2Fsingle-image-gen.png?alt=media) ## What are the best practices when dealing with Image generation ? Image generation can be a bit more difficult to get right compared to traditional text only tasks. It is good\ to follow a few rules for best results: * always explicitly mention that the goal is to generate one or multiple images in the instructions. This is\ especially required for models like Gemini Flash 2.0 Exp that handle more than just image generation. * use templated instructions (see [Variables in instructions](/python-sdk/agent#variables-in-instructions))\ to inline your input into the message sent to the LLM. Some models like\ GPT Image 1 are more tolerant to non plain text prompts but others like Imagen are really sensitive\ to non alphanumeric characters and will generate gibberish if prompted incorrectly. For example, for an agent that generates an image based on an animal (`string`) and a situation (`string`),\ your instructions could be: ``` Generate an image that represents an {{ animal }} in the situation: {{ situation }}. ``` ## How to customize image generation ? Image specific fields are automatically extracted from the instructions. For example, if your instructions\ mention that you prefer landscape images, the generated images will have a landscape format. Here are the variables that are extracted * `quality`: The quality of the image to generate. Can be `low`, `medium`, or `high`. * `background`: The background type for the image. Can be `opaque`, `transparent`, or `auto`. * `format`: The file type for the image. Can be `png`, `jpeg`, or `webp`. * `shape`: The shape of the image to generate. Can be `square`, `portrait`, or `landscape` {% hint style="info" %} Not all models support all parameters so models may return images that do not exactly match the desired\ configuration. {% endhint %} ## Generating multiple images It is possible to generate multiple images in a single call: * either by having multiple image fields in the output * or by having a list of images in the output. In this case, an `image_count` field should be added to the\ root of the input to control how many images are generated. ![Multiple images](https://2418444523-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FW4ng0K5LfFjYqHYuPgNh%2Fuploads%2Fgit-blob-bb83cde6de2c2ab2470352e0924c66ae6954a6e2%2Fmultiple-image.png?alt=media) ## How to edit existing Images ? An image editing feature is simply an image generation feature that contains an image as an input. An additional `mask` image field can be added to the input schema to allow masking the image. Not all model supporting image generation support image editing. ![Edited image](https://2418444523-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FW4ng0K5LfFjYqHYuPgNh%2Fuploads%2Fgit-blob-b6075da9267914d174d5e9756e8af85bf3bd7033%2Fimage-edit.png?alt=media) ## What about generated code ? The code corresponding to image generation features is very similar to the code for any other features.\ The only difference is that it contains an Image in the output. For example, in python: ```python from pydantic import BaseModel import workflowai from workflowai.fields import Image class ImageGenerationInput(BaseModel): prompt: str class ImageGenerationOutput(BaseModel): image: Image @workflowai.agent(id="image-generation", model="gpt-image-1") async def image_generation(_: ImageGenerationInput) -> ImageGenerationOutput: ... async def generate_image(prompt: str): run = await image_generation.run(ImageGenerationInput(prompt=prompt)) image = run.output.image print(image.content_type) print(image.data) # data is the base64 encoded image data ``` ## Can I create a feature that outputs an image AND text? Supposedly, generating image and text is currently supported by Gemini Flash 2.0 Exp. In practice, it can be\ quite difficult to get consistent results. If this is something you'd like to see us widely support, please add a feature request\ on [GitHub](https://github.com/workflowai/workflowai/discussions) or [Discord](https://discord.com/invite/qcsq8Kva). # Schemas ## What are schemas? An AI feature has at least one schema. Each schema define: * an input structure * an output structure For example, a [feature that answer question about a PDF](https://workflowai.dev/workflowai/tasks/pdf-question-answering/1/schemas) is represented: ![An example of a schema](https://2418444523-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FW4ng0K5LfFjYqHYuPgNh%2Fuploads%2Fgit-blob-9f7940c1402e6d276946dce951eb166b3fa777c7%2Fschema.png?alt=media) {% tabs %} {% tab title="Python" %} {% hint style="info" %} WorkflowAI uses [Pydantic](https://docs.pydantic.dev/) to define schemas. {% endhint %} ```python class PdfQuestionAnsweringTaskInput(BaseModel): pdf_document: Optional[File] = None question: Optional[str] = None class SupportingQuote(BaseModel): quote: Optional[str] = None page_number: Optional[float] = None class PdfQuestionAnsweringTaskOutput(BaseModel): answer: Optional[str] = None supporting_quotes: Optional[list[SupportingQuote]] = None ``` {% endtab %} {% tab title="TypeScript" %} ```typescript interface PdfQuestionAnsweringTaskInput { pdf_document?: File; question?: string; } interface SupportingQuote { quote?: string; page_number?: number; } interface PdfQuestionAnsweringTaskOutput { answer?: string; supporting_quotes?: SupportingQuote[]; } ``` {% endtab %} {% endtabs %} ## Examples For example, if you want an agent to summarize a text, the input is a text, and the output is a summary. If you want this agent to summarize a text in a specific language, you'll need to add a language parameter to the input. The inputs are like all the variables the LLM will have access to. The outputs are the different variables the LLM will generate. {% hint style="info" %} When using WorkflowAI web-app, you can write what you want the agent to do, and the web-app will generate a schema for you. {% endhint %} ## Why are schemas a good idea? Clear input and output structures (=schemas) have a few benefits: 1. simplify integration with a backend by providing a clear interface 2. provide output consistency 3. increase the quality of LLM outputs by structuring the task ### Technical details WorkflowAI leverages structured generation, also called [structured output](https://platform.openai.com/docs/guides/structured-outputs), or [controlled generation](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/control-generated-output). Structured generation is currently enabled for [all supported OpenAI models](https://platform.openai.com/docs/guides/structured-outputs), and for all models on [Fireworks](https://docs.fireworks.ai/structured-responses/structured-response-formatting#structured-response-modes). When structured generation is not available, WorkflowAI automatically falls back to [JSON mode](https://docs.anthropic.com/en/docs/test-and-evaluate/strengthen-guardrails/increase-consistency), and **always guarantees** the output will follow the output schema. ## How to create a schema? WorkflowAI supports two ways to create a schema: * using our web-app, using AI or manually. * [writing code directly (for Python)](/python-sdk/get-started). ### Field data types When creating a schema, you will notice that each field is assigned a data type. In most cases, the AI schema generator will choose the correct data type. However, you can also manually choose the data type. Here are the different data types you can choose from: * `string`: a sequence of characters, like letters, numbers, and symbols. * `boolean`: the value can only be true or false. * `number`: can be both whole numbers (integers) and numbers with decimal points (floats). * `list`: contains 1 or more entries of another data types (ex. a list of strings). The number of entries can vary in different task runs of the same task. * `enum`: the value can only be one of a predetermined, finite number of options. Similar to a boolean, but the options can be customized. * `object`: a single instance of a defined structure. The structure can contain multiple properties within it. * `date`: includes a year, a month, and a day. * `date-time`: includes a year, a month, a day, an hour, a minute, and a second. * `time`: includes an hour, a minute, and a second. * `timezone`: is a region that has the same standard time, defined by the difference in hours from Coordinated Universal Time (UTC). * `html`: content formatted as an HTML page. * `image`: an image. Supported formats are PNG, JPG, and WEBP. * `audio`: an audio file. Supported formats are MP3, WAV, AIFF, AAC, OGG Vorbis, and FLAC. * `document`: an uploaded document. Supported formats are PDF, CSV, TXT, and supported image formats. ## Edit a schema Finding the right schema takes a few iterations, so we try to make editing a schema as easy as possible. {% embed url="" %} When possible, we recommend to edit the schema using the agent via the playground. If you need more control, you can manually edit the schema. {% embed url="" %} ## Archiving and restoring a schema When building a new task, it's very likely you'll need multiple iterations to get the right schema. To clean up unused schemas, you can archive them. To archive a schema, navigate to the "Schemas" section from the menu, and click on the "Archive" button in the schema's detail view. To restore a schema, navigate to the "Schemas" section from the menu, and click on the "Restore" button in the schema's detail view. {% embed url="" %} {% hint style="info" %} Archived schemas are not deleted, but hidden from the UI. Any deployment or version using an archived schema will continue to work, to avoid breaking changes. {% endhint %} # Versions ## What is a version? A version is a specific **configuration** of an agent. WorkflowAI defines two types of (agent) versions: | Version Type | Example | Description | | ------------------ | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------ | | **Major** Versions | 1, 2, 3, ... | A major version represents a specific configuration of a agent, including its instructions, temperature, descriptions/examples, and tools. | | **Minor** Versions | 1.1, 1.2, 1.3, ... | A minor version represents a major version **associated with a specific model** (e.g., OpenAI's GPT-4o-mini). | ![Version 2 is a major version, version 2.1 is version 2 running on Gemini 2.0](https://2418444523-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FW4ng0K5LfFjYqHYuPgNh%2Fuploads%2Fgit-blob-dd71ee668a040738cd565059520b3f262e161694%2Fversions.png?alt=media) ## Why are versions useful? Versions are useful for several reasons: * They allow you to save a specific configuration of an agent, so you can reproduce it later. * They allow you to [compare the performance of different versions of an agent](/ai-features-playbook/evaluating-your-ai-feature). * They allow you to [deploy a specific version of an agent](/features/deployments). ## How to: ### Save a version When using the playground, you can save a version by clicking on the "Save" button. ![Save a version](https://2418444523-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FW4ng0K5LfFjYqHYuPgNh%2Fuploads%2Fgit-blob-feb4d4bef0467b8d7bc262618d101c0ef9e6bd8e%2Fsave-version.png?alt=media) Additionnaly, you can save all versions currently running by clicking on the "Save all versions" button. ![Save all versions](https://2418444523-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FW4ng0K5LfFjYqHYuPgNh%2Fuploads%2Fgit-blob-532117833f3902d81ded6dd642e61d96dd2b0796%2Fsave-all-versions.png?alt=media) {% hint style="info" %} You should save a version when you're satisfied with the LLM output and want to preserve the current configuration. A saved version captures your chosen model, prompt, temperature setting, and other parameters, allowing you to reliably reproduce these results later, or use them in a [deployment](https://github.com/WorkflowAI/documentation/blob/main/deployments/README.md). {% endhint %} ### List all versions You can access the list of all versions by clicking on the "Versions" section from the menu. ![List of all versions](https://2418444523-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FW4ng0K5LfFjYqHYuPgNh%2Fuploads%2Fgit-blob-db9ad237eaff4b24e0c07b7db0b9afc966f2d399%2Fversions-section-full.png?alt=media) ### Clone a version Cloning a version is useful when you want to create a new version based on an existing version. For example, you have a version running on OpenAI's GPT-4o-mini, and you want to quickly reuse the same instructions, temperature, and tools for a new version running on Gemini 2.0. {% embed url="" %} # Runs A run is a single execution of an agent. For example: ![Run view](https://2418444523-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FW4ng0K5LfFjYqHYuPgNh%2Fuploads%2Fgit-blob-de7d301b42cbd6ec9498a6a5e67072ba1bbc2c7d%2Frun-view.png?alt=media) Each run has a unique identifier and can be accessed directly via a URL, like [this example run](https://workflowai.com/docs/agents/review-summary-generator/runs/0195dd7a-6977-7197-7ec3-4fc44ade50dc). {% hint style="warning" %} **Privacy Note**: Run URLs are private by default and only accessible to users within your organization. They are not publicly accessible, ensuring your data and AI interactions remain secure. {% endhint %} By default, WorkflowAI stores all runs, available in the "Runs" section. You can view a list of all runs for a specific agent, like [this example runs list](https://workflowai.com/docs/agents/review-summary-generator/1/runs?page=0). ![Run list](https://2418444523-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FW4ng0K5LfFjYqHYuPgNh%2Fuploads%2Fgit-blob-37a529b73871152aa5ecb98a5b2816bb94423e58%2Flist-runs.png?alt=media) ## Why storing all runs? * **Observability**: Understand how the AI is performing by tracking and analyzing its outputs over time. * **Saving cost**: For the same input and model versions, cached runs can be served without triggering a new LLM call, reducing costs to $0 for serving cached runs. * **Fine-Tuning and distillation**: Saving all runs is required for fine-tuning models and distillation processes. ## How to search runs? WorkflowAI provides a powerful search – available under the "Runs" section – to find specific runs: {% embed url="" %} {% hint style="info" %} **Architecture**: Under the hood, runs are stored in a Clickhouse database, which is optimized for handling large amounts of data, and for fast search and aggregation queries. Clickhouse also compresses data, which reduces storage costs. Learn more about Clickhouse [here](https://clickhouse.com/docs/en/introduction). {% endhint %} ## View a run's prompt and response WorkflowAI provides full transparency into the interaction with the LLM. You can easily examine both the raw prompt sent to the model and the complete response received: 1. Navigate to any run's detail view 2. Click the "View Prompt" button to see the exact instructions sent to the LLM You can try viewing the prompt for [this example run](https://workflowai.com/docs/agents/review-summary-generator/runs/0195dd7a-6977-7197-7ec3-4fc44ade50dc). {% embed url="" %} ## Try in playground To import a run into the playground, you can use either: * the "Try in playground" button in the run detail view, which will automatically import the run input, and the version used to generate the run. * only import the run input, by clicking on the "Try Input in Playground" button in the run detail view. {% embed url="" %} ## Delete a run Deleting a specific run is not possible. However, you can delete all runs for a specific agent, by deleting the agent. # Tools ## What are tools? Tools enable AI agents to: | | | | ------------------------ | ------------------------------------------------------------- | | **Access External Data** | Web searches, web scraping, databases, APIs, files | | **Perform Actions** | Form submissions, code execution, API calls, custom functions | Tools have two forms: | **Custom Tools** | Developer-defined tools. Custom tools will require you to write code to handle the tool calls. | | ---------------- | ------------------------------------------------------------------------------------------------------------------------ | | **Hosted Tools** | WorkflowAI-built tools (e.g., *search-web*, *browser-text*). Hosted tools do not require any code or engineering effort. | ## Custom Tools Custom tools are tools specific to your application. You are responsible for running these tools when they are called by the AI agent. Adding a custom tool through code is currently only available with our Python SDK. Read the documentation for [adding custom tools](/python-sdk/tools). ## Hosted Tools ### What hosted tools are available? WorklfowAI supports and manages the following tools: * `@browser-text` allows fetching the content of a web page (text-only) * `@google-search` allows performing a web search using Google's search API * `@perplexity-sonar-pro` allows performing a web search using Perplexity's Sonar Pro model We're working on adding more tools, if you need any specfic tool, please open a discussion on [GitHub](https://github.com/workflowai/workflowai/discussions/categories/ideas) or [Discord](https://discord.gg/auuf8DREZh) ### How to enable tools? Tools can be added in the Playground by either: 1. Describing the use case to the playground chat agent 2. Under "Instructions" tap on the tools you want to enable. {% embed url="" %} # Playground The playground is WorkflowAI's tool to allow quick and easy iterating on prompts and models, so you can create the best version of your AI feature possible. ![Playground](https://2418444523-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FW4ng0K5LfFjYqHYuPgNh%2Fuploads%2Fgit-blob-b8c7dbcc28d019f1cb59734aea9ded72e8d70313%2Fplayground-fullscreen.png?alt=media) ## What are the different parts of the playground? The playground is composed of 3 sections: * **Input**: where the content given to the AI feature is defined. * **Parameters**: where the details of how the AI feature should behave are described. * **Outputs**: where the AI feature's results are displayed. ### Input You can manually enter the input content, import existing data, or - for text inputs only - you can use WorkflowAI to generate synthetic data. {% embed url="" %} ### Parameters Parameters dictate how the AI feature should behave. #### Prompt Instructions The prompt are the instructions given to the LLM to describe what is expected from the agent and how it should process the input. When you create a new agent, WorkflowAI will generate a default prompt for you. You can manually edit the prompt to refine the behavior of the AI feature, but our recommended approach is to use the chat agent on the playground to help with editing. Simply describe the problem with the current behavior or what you want to see instead, and the chat agent will help iterate on the prompt to get the results you're looking for. #### Temperature Temperature is a parameter that controls the randomness of the LLM's outputs. WorkflowAI provides three preset temperature settings: * **Precise** (Default): The recommended and default setting for all tasks * Best for tasks requiring accuracy and consistency * Ideal for factual responses, code generation, or structured data extraction * Produces reliable, repeatable results * **Balanced**: Moderate setting that balances creativity and coherence * Good for general-purpose tasks * Works well for most conversational and analytical tasks * Provides reasonable variation while maintaining relevance * **Creative**: Maximum diversity and exploration * Best for tasks requiring unique or innovative outputs * Ideal for brainstorming, creative writing, or generating alternatives * Produces more varied but potentially less focused results * **Custom**: User-defined temperature setting * Allows precise control over the temperature value * For advanced users who understand the impact of temperature "Precise" is automatically selected as the default temperature setting for all new features to ensure consistent and reliable outputs. You can adjust this in the Parameters section of the Playground if your use case requires more variation. ![Prompt and Temperature](https://2418444523-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FW4ng0K5LfFjYqHYuPgNh%2Fuploads%2Fgit-blob-f499a4624504ed0d732e9062c8b0373f76e57f67%2Fparameters.png?alt=media) #### Descriptions and Examples Descriptions and Examples are optional fields available for string-based fields that can be used to provide additional, field-specific information to the LLM. If you find that the prompt instructions alone are not guiding the LLM to handle a specific field, you can add or modify the description or examples to help the LLM understand the field better. * **Description**: A description of the field can including what it is and/or what it is used for. * **Examples**: You can add one or multiple examples of a field to help the LLM understand what the field formatting should look like. Examples and Descriptions edits are made on the playground by: * **In all cases:** Asking the playground agent to make an update on your behalf by describing the change * **If output is currently displayed:** Hovering over a field name, then hover over the description/examples modal and tap Edit * **If no output is displayed:** Hovering over a description or example and tap Edit {% embed url="" %} ### Outputs The outputs section displays the LLM outputs. For each output, WorkflowAI also displays: * 💰 Cost: The total price in USD for generating this output * ⚡ Latency: How long it took to get a response from the model, in seconds * 📊 Context window usage: How much of the model's maximum token limit was used, shown as a percentage > WorkflowAI Cloud offers a price-match guarantee, meaning that you're not charged more than the price per token of the model you're using. Learn more about the price-match guarantee [here](https://workflowai.com/pricing). {% embed url="" %} ### Diff mode You can enable diff mode to highlight the differences between LLM outputs, making it easy to spot differences in how models handle your task. Diff mode can be especially helpful for text-heavy outputs, like texts summarizations or composition. {% embed url="" %} # Reviews Reviewing runs is a way to evaluate whether an individual run's output is correct or incorrect based on the given input. There are two types of reviews: * **Human-Reviewed:** a human reviewer manually evaluates a run and marks it as correct or incorrect. ![Human Review](https://2418444523-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FW4ng0K5LfFjYqHYuPgNh%2Fuploads%2Fgit-blob-474912f01bd6960dea36570a21a3c82f8b27affe%2Fhuman-review.png?alt=media) * **AI-Reviewed:** an AI agent evaluates a run and marks it as correct or incorrect. AI reviews require a human review on the same input in order to run, as the human review is used as a baseline for the AI review. ![AI Review](https://2418444523-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FW4ng0K5LfFjYqHYuPgNh%2Fuploads%2Fgit-blob-983cc3bc3e26b77ef733185e51efe0c4ca71561d%2Fai-review.png?alt=media) ## Why is reviewing runs important? When you leave a review on a run, you're adding that input/output pair - and whether is it correct - to your Reviews dataset. This dataset is used to benchmark the performance of different versions of your feature. The more runs that are evaluated, the larger your Reviews dataset will be, which will create a more accurate result for your benchmarks. [Learn more about how benchmarking works.](/features/benchmarks) Additionally, the more reviews that are added, the more the AI reviewer agent will understand all the criteria that determines correctness for your feature, and it will be able to handle evaluating runs on your behalf more effectively. ## How do I review runs? Before you can leave any reviews, you have to run your AI feature first. You can create runs from the [Playground](/features/playground) or from our [Python SDK](/python-sdk/get-started). Runs can be reviewed in two places: * **Playground:** if the feature is easy to test using generated inputs, runs can be reviewed on the playground as they’re completed. Just locate the green thumbs up and red thumbs down icon under the run output, and select the appropriate option. * **Runs Page:** if runs are coming from the API/SDK (or if you want to add a review from the playground after the fact), runs can be reviewed any time after they’re completed from the Runs page. To review a run, locate the run and select it to open the run details page. Then, select the green thumbs up or red thumbs down icon under the run output to add a review. {% embed url="" %} ## Why do some of the review icons look different? There are two types of reviews: * **Human-Review:** a human reviewer manually evaluates a run and marks it as correct or incorrect. When a human review is left, you will see that the corresponding icon and background are solid. ![Human Review](https://2418444523-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FW4ng0K5LfFjYqHYuPgNh%2Fuploads%2Fgit-blob-474912f01bd6960dea36570a21a3c82f8b27affe%2Fhuman-review.png?alt=media) * **AI-Powered Review:** an AI agent evaluates a run and marks it as correct or incorrect. AI-powered reviews require an initial human review on the same input as a baseline for the AI review. When an AI review is left, you will see that the corresponding icon is highlighted, but not filled in. AI reviews can always be overriden by a human reviewer, so if something does not look right, you can correct the review. ![AI Review](https://2418444523-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FW4ng0K5LfFjYqHYuPgNh%2Fuploads%2Fgit-blob-983cc3bc3e26b77ef733185e51efe0c4ca71561d%2Fai-review.png?alt=media) {% hint style="info" %} **What's the point of AI-powered reviews?** AI-powered reviews save time by automatically reviewing runs with an input that matches an already human-reviewed run. So when you're iterating on your prompt with the same input, you won't need to manually review every single run. AI-powered reviews are also used to calculate accuracy scores for versions when benchmarking. [Learn more about how benchmarking works.](/features/benchmarks) {% endhint %} ## Can more than one output be considered correct? In many cases, yes. For example, if there is a feature meant to summarize an article, there may be multiple summaries that contain all the correct information, but are phrased differently. In these cases, it's helpful to leave a review on all correct outputs. The multiple reviews help the AI reviewer agent get a better understanding of all the critieria that determines correctness. ## How big should my evaluation dataset be? How many runs should I review? In most cases, we recommend reviewing 10-20 separate inputs to build an initial dataset. However the actual number may vary depending on your use case. The more complicated your feature is, the more reviews should be added to ensure that you have a robust dataset that covers all your important use cases. # Side by Side Side by side is a minimal-setup-required feature that allows you to quickly compare how two different versions or models handle a different inputs. ## How to use Side by Side While the goal of side by side is to be as minimal-setup-required as possible, there are a few prerequisites: * You need to have at least one version of a feature saved. * You need to have at least one input to compare, although we recommend having a minimum of 10 to provide a robust dataset to generate a comparison. One you meet the above requirements, to use Side by Side: 1. Login to [WorkflowAI](https://workflowai.com/) 2. Select the AI feature and you want to compare 3. Visit the Side by Side page in the sidebar 4. In the left column, you will be able to see the inputs that will be used to generate the comparison. 5. In the middle column, select the currently existing version of the feature you want to compare against (if you have a version of your AI feature deployed, you'd likely want to select the deployed version here). 6. In the right column, select the new model or version of the feature you want to compare against the existing version. Once you have selected the two versions, it may take a moment to run the inputs through the two versions and generate a comparison, but onces that's done, you'll be able to see how the two different versions handle the same inputs, side by side for easy comparison. {% hint style="info" %} This feature is still in it's early stages of development, so more functionality will be added in the future. If there's something you'd like to see, please let us know on [GitHub](https://github.com/workflowai/workflowai/discussions) or [Discord](https://discord.gg/auuf8DREZh). {% endhint %} {% embed url="" %} ## What's the difference between Side-by-Side and Benchmarks? In short: side by side is better at giving you a quick vibe check of a new version or model, while benchmarking is better at providing a more thorough evaluation across multiple versions. With Side by Side, you can quickly compare the outputs of two different versions of an AI feature, but since side by side doesn't currently include any AI powered analysis or reviews, you will need to manually assess which version you think performs better. Side by side is also limited to comparing two versions at a time, meaning that you will to have a sense of which versions you want to compare against before you can use side by side. Benchmarking, on the other hand, does include AI powered analysis and reviews to gauge which model is more accurate. However, in order to ensure that a benchmark generates a fair and useful accuracy score, it's important that you spend the time to build up a large enough evaluation dataset. See [How big should my evaluation dataset be/how many runs should I review?](#how-big-should-my-evaluation-dataset-behow-many-runs-should-i-review) for more information. Because of this need for a dataset, benchmarking is a more time-intensive process than side by side. Benchmarking also allows you to compare multiple versions at once, meaning that if you don't yet known which two versions or models you want to compare, you can simultaneously benchmark as many versions as you want and see which one(s) stand out. ## Can I use Side by Side when a new model comes out? Yes! We built Side by Side with this use case in mind. You can use Side by Side to get a quick vibe check of a new model compared to your currently deployed model. All you need to do is navigate to the Side by Side page and select the new model from the dropdown menu on the right. # Benchmarks Benchmarks are a way to find the best version of your agent based on a quantitative comparison of the performance, cost and latency of each version. ![Benchmarks](https://2418444523-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FW4ng0K5LfFjYqHYuPgNh%2Fuploads%2Fgit-blob-3b9024db975cfe8e38019de701d2e82303c9153c%2Fbenchmark-table.png?alt=media) ## How do I benchmark an AI Feature? In order to benchmark an AI Feature, you need to have two things: * Reviewed runs (we recommend starting with between 10-20 reviews, depending on the complexity of your AI Feature). You can learn more about how to review runs [here](/features/reviews). * At least two saved versions of the AI Feature on the same Schema. To save a version, locate a run on the Playground or Runs page and select the "Save" button. This will save the parameters (instructions, temperature, etc.) and model combination used for that run. After creating a review dataset and saving versions of your AI feature that you want to benchmark, access the Benchmark page in WorkflowAI's sidebar and select the versions you want to compare. The content of your review dataset will automatically be applied to all selected versions to ensure that they are all evaluated using the same criteria. {% embed url="" %} ## How exactly does benchmarking work? ### Accuracy Version accuracy is based off of the human reviews left on runs and supplimented by AI-powered reviews. The process goes as such: 1. AI feature runs are given reviews by a human reviewer. The review runs are added to the Reviews dataset page (visible on the [Reviews page](/features/reviews)). 2. When benchmarking a version, the selected version runs all the inputs present in the Reviews dataset. 3. Using the human reviews from the dataset as a baseline, the AI-powered reviews are added to evaluate any runs on the benchmarked version that don't yet have a human review. 4. The amount of correct and incorrect runs - based on both the human reviews and AI-powered reviews - are used to calculate the accuracy of the version. ### Price Price is calculated based on the number of tokens used in the version's runs and cost of the tokens for the selected model. ### Latency Latency is calculated based on the time it takes for each of the version's runs to complete. ## How can I add a specific input to my benchmark evaluation dataset? If there are inputs that you want to ensure are included when benchmarking a version, all you need to do is review at least one run of that input. Once an input has been reviewed, it will be automatically be added to your Reviews dataset and ultilized when benchmarking. ## A new model was released, how can I quickly evaluate it? We're actively working on an even faster way to evaluate new models, but in the meantime here is the current process we recommend: To quickly benchmark a new model: 1. Locate the feature you want to test the new model on 2. Make sure the schema selected in the header matches the schema you have deployed currently. 3. Go to the versions page and locate your currently deployed version (you will recognize it by the environment icon(s) next to the number) 4. Hover over the version and select **Clone** and then select the new model. 5. Go to the benchmark page and select both the currently deployed version and the new version you just created. Note: in order to ensure that a benchmark generates a fair and accurate comparison, it's important that you have a large enough evaluation dataset. See [How big should my evaluation dataset be/how many runs should I review?](/ai-features-playbook/evaluating-your-ai-feature#how-big-should-my-evaluation-dataset-be-how-many-runs-should-i-review) for more information. # Code ## How do I integrate a task in my codebase? {% hint style="success" %} For software engineers {% endhint %} We try to make it as easy as possible to integrate an AI Feature into your codebase. 1. Go to the Code page for your feature. 2. Select the coding language you want to use. Currently we support generating code for Python, Typescript, and REST API. 3. Select the version you want to use. * We highly recommend deploying a version to an environment before integrating it into your codebase. This way, your generated code will reference an environment variable instead of a hardcoded version number, allowing you to update the version without breaking changes. 4. If you have not already, install the WorkflowAI package, using the command provided on the code page. 5. Copy the code snippet and paste it into your codebase. 6. Create a secret key and paste it into the code snippet in your codebase. {% embed url="" %} ## Caching | Option | Description | | ---------------- | --------------------------------------------------------------------------------------- | | `auto` (default) | Completions are cached only if they have `temperature=precise` (or `0`) | | `always` | Completions are always cached, even if `temperature` is set to `Balanced` or `Creative` | | `never` | The cache is never read or written to | {% hint style="warning" %} Even with `cache=never`, using `temperature=precise` will still produce consistent outputs because the AI model itself is deterministic at this setting. To get varied outputs, change the temperature to `Balanced` or `Creative` (or any value greater than 0). {% endhint %} ## Streaming You can enable result streaming from your AI Feature to reduce latency and enhance responsiveness. This is particularly useful for user-facing interactions, where shorter wait times significantly improve the overall user experience, since streaming ensures users see results progressively, rather than waiting for the entire output to load at once. To enable streaming for your AI Feature: 1. Go to the **Code** page for your feature 2. Select the version and coding language you'd like to use in your product 3. Under the **Streaming** section, confirm that streaming is enabled before copying and pasting the generated code snippet into your codebase. # Deployments Deploy specific versions of an agent with ease, allowing for updates to prompts and models **without any code changes**. ## Why use deployments? * ✅ update to a new model or prompt without asking your engineering team. * ✅ save cost by updating to a more recent, cheaper model, without changing your code. * ✅ improve the quality of your tasks outputs by adjusting the prompt, in real-time, based on users' feedback. * ✅ use different versions of a task in different environments (development, staging, production). ## How to deploy a version? 1. Go to **Deployments** section from the menu. 2. Pick the environment you want to deploy to, either: production, staging, or development. 3. Tap **Deploy Version** 4. Select the version you want to deploy. 5. Tap **Deploy** {% embed url="" %} {% hint style="warning" %} To avoid any breaking changes: deployments are **schema specific**, not AI feature specific. This means that if you want to deploy a new version of your AI feature that is on a different schema, you will need to update the schema number in your code. This also means that you can deploy a (development, staging, production) version for each schema of an agent without the version deployed to production being affected. {% endhint %} ## Using your own AI Provider Keys Your own API provider keys can be added by going to: [workflowai.com/organization/settings/providers](https://workflowai.com/organization/settings/providers). **Important: If you are using WorkflowAI Cloud, credits will still deducted by default. There is a required manual operation on our side to mark the keys as a customer provided key.** If you would like to use your own keys via workflowai.com, please reach out to us via [email](mailto:team@workflowai.support) or on [GitHub](https://github.com/workflowai/workflowai/discussions) so we can help you out. If you are self-hosting, adding your own API provider keys does not require any additional steps beyond adding them at: [workflowai.com/organization/settings/providers](https://workflowai.com/organization/settings/providers). # User Feedback WorkflowAI allows you to collect feedback from users about your AI features. {% hint style="info" %} Collecting end user feedback is essential for understanding how your AI features perform in real-world scenarios. The main goal is to gather insights about user satisfaction and feature effectiveness when users interact with your AI agents in production environments. This data helps you identify strengths and weaknesses, prioritize improvements, and measure the overall health of your AI-powered features over time. {% endhint %} ## Feedback loop 1. Add a feedback button to your product: using our web SDK, or by using our API. 2. Users click the button and give feedback. 3. Your team can view the feedback in the WorkflowAI dashboard. 4. Improve instructions based on the feedback. ## How it works ### Feedback Token Lifecycle The feedback system operates through a secure `feedback_token` that links user feedback to specific AI interactions: 1. **Token Generation**: When you call the `/run` endpoint to execute an AI agent, WorkflowAI automatically generates a unique `feedback_token` for that specific interaction. 2. **Token Security**: The `feedback_token` is a cryptographically signed token that: * Is valid only for the specific run that generated it * Cannot be used to access any sensitive data * Requires no additional authentication to submit feedback 3. **Token Propagation**: Your application needs to pass this token from your backend to your frontend client application where feedback will be collected. 4. **Feedback Submission**: When a user provides feedback, your application sends the `feedback_token` along with the feedback data (positive/negative rating and optional comment) to WorkflowAI. 5. **Storage and Analysis**: WorkflowAI associates the feedback with the original run, making it available in your dashboard for analysis. {% hint style="info" %} The `feedback_token` is designed to be safely passed to client-side applications. It contains no sensitive information and can only be used for submitting feedback for the specific run that generated it. The token cannot be used to access any user data, modify your agents, or perform any administrative actions. This security-by-design approach allows you to freely incorporate feedback collection in your frontend without compromising your application's security. {% endhint %} ### User ID Tracking * The optional `user_id` parameter allows tracking feedback on a per-user basis * Each unique combination of (`feedback_token`, `user_id`) can have only one feedback entry * Submitting new feedback with the same (`feedback_token`, `user_id`) pair will overwrite previous feedback * This prevents duplicate feedback while allowing users to change their minds ### Data Flow Diagram ``` Backend Frontend WorkflowAI ┌────────────┐ ┌────────────┐ ┌────────────┐ │ │ 1. Call /run API │ │ │ │ │ │───────────────────────────────────────────────────────────> │ │ │ │ │ │ │ │ │ 2. Receive token │ │ │ │ │ Your │<─────────────────────────────────────────────────────────── │ │ Server │ │ Your │ │ WorkflowAI │ │ │ 3. Pass token │ Client App │ │ API │ │ │───────────────────────> │ │ │ │ │ │ │ 4. Submit feedback │ │ │ │ │ │───────────────────────> │ │ │ │ │ │ │ └────────────┘ └────────────┘ └────────────┘ ``` ## Access `feedback_token` `feedback_token` needs to be accessed from the client application that will be used to post feedback. ### Python SDK ```python import workflowai # Get feedback token from run run = await my_agent.run(MyAgentInput()) print(run.feedback_token) # Get feedback token when streaming async for chunk in my_agent.stream(MyAgentInput()): # Process chunks pass print(chunk.feedback_token) ``` ### Typescript SDK ```typescript import { WorkflowAI } from "@workflowai/workflowai"; const workflowAI = WorkflowAI(); // Get feedback token from run const { output, feedbackToken } = await myAgentFunction(input); console.log(feedbackToken); // Get feedback token when streaming let lastChunk: RunStreamEvent | undefined; for await (const chunk of stream) { lastChunk = chunk; } console.log(lastChunk?.feedbackToken); ``` ### API The feedback token is returned by the run endpoint. See the [endpoint documentation](https://run.workflowai.com/docs#/Run/run_task_v1__tenant__agents__task_id__schemas__task_schema_id__run_post). ``` POST /v1/_/agents/my-agent/schemas/1/run Host: https://run.workflowai.com Authorization: Bearer {Add your API key here} Content-Type: application/json # JSON Body { "task_input": ... } # Response { "task_output": ..., "feedback_token": ... } ``` ## Post feedback ### Web SDK The web SDK is the simplest way to add a feedback button to your web app. #### React ```bash npm install @workflowai/react ``` ```typescript import { FeedbackButtons } from '@workflowai/react' ... ... ``` ### SDKs/REST API Use our API if: * you want full customization over the feedback button and send the feedback via your own backend. * you want to post feedback from a non-browser environment (e.g. mobile apps). #### Python ```python import workflowai await workflowai.send_feedback(feedback_token="...", outcome="positive", comment=..., user_id=...) ``` #### Typescript ```typescript import { WorkflowAI } from "@workflowai/workflowai"; const workflowAI = WorkflowAI() await workflowAI.sendFeeback({feedback_token: "", outcome: "positive", comment: "...", userID: ""}) ``` #### REST API Posting feedback is a single non authenticated API call with a `feedback_token` and `outcome` in the body.\ See the [full documentation](https://api.workflowai.com/docs#/Feedback/create_run_feedback_v1_feedback_post). ``` POST /v1/feedback Host: https://api.workflowai.com Content-Type: application/json { "feedback_token": "...", # the token as returned by the run endpoint "outcome": "positive", # "positive" | "negative" "comment": "...", # optional, the comment from the user "user_id": "..." # optional, if provided, feedback will be associated with a specific user. Posting feedback for the same `feedback_token` and `user_id` will overwrite the existing feedback. } ``` ## View user feedback Go to the "User Feedbacks" section from the menu, and you'll see a list of feedback. ![User Feedback Screen](https://2418444523-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FW4ng0K5LfFjYqHYuPgNh%2Fuploads%2Fgit-blob-b55548f3121be7bcb25d62eb7f8260f86cb8d8e5%2Fuser-feedback.png?alt=media) {% hint style="info" %} If you need any help, email us at or open a discussion on [GitHub](https://github.com/workflowai/workflowai/discussions) or [Discord](https://discord.gg/auuf8DREZh) {% endhint %} # Monitoring An overview of each AI features run count and cost for the past week is available on your [home page](https://workflowai.com/), once logged in. To get further insights to how an AI feature is performing, you can go to the "Cost" section from the sidebar menu, to access a dashboard with the number of runs, and cost metrics for additional time frames. ![Monitoring](https://2418444523-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FW4ng0K5LfFjYqHYuPgNh%2Fuploads%2Fgit-blob-24932acae8ae8506f01d9e1c8c3c0ec3cc39938c%2Fmonitoring.png?alt=media) More metrics, including average feature latency and organization-wide monitoring , are coming soon! {% hint style="info" %} If you need more metrics, please open a discussion on [GitHub](https://github.com/workflowai/workflowai/discussions/categories/ideas) or [Discord](https://discord.gg/auuf8DREZh) {% endhint %} # Limitations Currently, WorkflowAI does not support: * listening and participating in real-time audio conversations. However AI features can understand audio files as input. * fine-tuning and distillation. But you can export the outputs of an agent from WorkflowAI and use them in your own fine-tuning pipeline. {% hint style="info" %} We expect that all these use cases will be supported in the future (by end of 2025). {% endhint %} # Change Log *** ## Release Notes April 29, 2025 **TL;DR: New image generation capabilities with three powerful models (GPT-Image, Gemini Flash Experimental, Imagen 3), plus faster and more cost-effective Llama 4 Maverick and Scout models.** **Generate Stunning Visuals with Cutting-Edge Image Models** WorkflowAI now empowers you to generate images directly within the platform, opening up a world of creative possibilities. Access the same powerful technology that took the internet by storm, with viral trends like the Studio Ghibli-style transformations that allowed users to "Ghiblify" their photos and memes. This capability underscores WorkflowAI's commitment to providing access to state-of-the-art AI models, all in one place, without the complexity of managing multiple integrations. We're introducing three new models to kickstart your image generation journey: * GPT-Image from OpenAI: This versatile model, the same one powering ChatGPT's image generation, excels at creating images across diverse styles, faithfully following custom guidelines, leveraging world knowledge, and accurately rendering text. Pricing is $5 per 1M text input tokens, $10 per 1M image input tokens, and $40 per 1M image output tokens, translating to roughly $0.02 to $0.19 per generated image depending on quality. * Gemini 2.0 Flash Experimental: Generate text and inline images conversationally with Gemini. Edit images or create outputs with interwoven text, like blog posts with integrated visuals. This experimental model is currently free to use, allowing you to explore its capabilities without cost. Note that all generated images include a SynthID watermark. * Imagen 3: Generate images with exceptional detail, rich lighting, and fewer artifacts. Imagen 3 understands prompts written in natural language, generates images in various formats and styles, and renders text effectively. At just $0.04 per image, Imagen 3 offers a cost-effective solution for high-quality image generation. ![](https://2418444523-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FW4ng0K5LfFjYqHYuPgNh%2Fuploads%2Fgit-blob-b9ced5c433f84b54fb4e5ba9d48c681bb0d045ba%2Fimage-side-by-side.png?alt=media) **Enhanced Llama Models for Optimal Performance** We've updated our Llama models to provide you with even better performance and value. The new Llama 4 Maverick and Llama 4 Scout models are cheaper and faster than the original versions we support, ensuring you get the most out of your AI workflows. You can differentiate the new models by the presence of a ⚡️ icon after their name in the playground. This update reflects WorkflowAI's dedication to benchmarking accuracy, speed, and cost, helping you select the best model with confidence. ![](https://2418444523-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FW4ng0K5LfFjYqHYuPgNh%2Fuploads%2Fgit-blob-6574ef6f9f6e1ae91b1b252f074cbc48184f66ef%2Fllama-side-by-side.png?alt=media) *** Ready to make your AI dreams a reality? Give our GitHub repo a star! It's like giving us a virtual high-five, but with more code and less sweat. [github.com/workflowai/workflowai](http://github.com/workflowai/workflowai) *These release notes were crafted by* [*WorkflowAI's Release Note AI Assistant*](https://workflowai.com/docs/agents/release-note-generation/1?showDiffMode=false\&show2ColumnLayout=false) *** ## Release Notes April 24, 2025 ### Introducing Gemini 2.5 Flash: Expanding Our Model Ecosystem At WorkflowAI, we're committed to providing access to the world's top AI models in one place. Today, we're excited to add Google's Gemini 2.5 Flash to our growing collection of state-of-the-art models. This addition reinforces our mission to bring you the best models from every provider—OpenAI, Anthropic, Google, Llama, Grok, and DeepSeek—all in one unified platform with zero setup required. Gemini 2.5 Flash offers an exceptional performance-to-cost ratio, placing it on the efficiency frontier. As a fully hybrid reasoning model, it allows you to toggle thinking capabilities on or off (in WorkflowAI, we have two separate models - one with thinking on, one without). **Pricing:** * Input tokens: $0.15/1M tokens (50% increase from Gemini 2.0 Flash) * Non-thinking output tokens: $0.60/1M tokens (50% increase) * Thinking output tokens: $3.50/1M tokens (775% increase) **Performance:**\ Google's benchmarks show Gemini 2.5 Flash outperforming its predecessor in nearly every category, including reasoning, mathematics, and code generation—all accessible through WorkflowAI's simple, unified API. ### New Side-by-Side Comparison: Make Confident Model Decisions No more switching between tabs and spreadsheets to compare models! WorkflowAI's new side-by-side view empowers you to directly compare any two models, helping you evaluate quality, cost, and speed to make data-driven decisions for your AI features. Wondering how Gemini 2.5 Flash performs against your current model? Simply navigate to the Side by Side page under the COMPARE section of any agent, and select Gemini 2.5 Flash—or any other model you're curious about—from the righthand column. This feature embodies our commitment to helping you find the best model for your specific use case, validating your AI decisions with real, comparable outputs. ![](https://github.com/WorkflowAI/documentation/blob/main/docs/docs/assets/images/release-note-assets/sidebyside.png) ### Join Our New Discord Community: Connect, Learn, and Grow Open source is at the heart of WorkflowAI, and building a strong community is essential to our mission. We're thrilled to announce the launch of our official Discord server—the perfect place to ask questions, share feedback, and connect directly with our team and fellow WorkflowAI users. This new community platform reinforces our commitment to transparency and collaboration. Whether you're looking to troubleshoot an integration, share your latest AI feature built with WorkflowAI, or simply connect with like-minded professionals, our Discord server is now your go-to destination. Join our Discord community: *** ⭐ If our repository were a model, it would need your stars to think properly! Help us reach astronomical intelligence by starring our GitHub repo: github.com/workflowai/workflowai ⭐ *These release notes were crafted by* [*WorkflowAI's Release Note AI Assistant*](https://workflowai.com/docs/agents/release-note-generation/1?showDiffMode=false\&show2ColumnLayout=false) # Introduction WorkflowAI Cloud is a fully managed version of WorkflowAI, maintained by the WorkflowAI team. ## When to use WorkflowAI Cloud? We recommend using WorkflowAI Cloud unless you have strict requirements on data-retention, data-residency, or compliance. For example, WorkflowAI Cloud is hosted only in the US at the moment, so if you want to have your data hosted in the EU, you should use WorkflowAI as a [self-hosted solution](https://github.com/workflowai/workflowai). ## Benefits of WorkflowAI Cloud The benefits of using WorkflowAI Cloud versus our own deployment are: * ✅ Fully managed: we take care of the infrastructure, so you can focus on building your AI agents. * ✅ Always up-to-date: we ensure that you're always using the latest stable version of WorkflowAI. * ✅ Cheaper: we offer a price-match guarantee, meaning that you're not charged more than the price per token of the model you're using. Learn more about the [price-match guarantee](/workflowai-cloud/pricing). Effectively cheaper than self-hosting. * ✅ Reliability: we ensure that your AI agents are [always available](/workflowai-cloud/reliability) * ✅ Compliance: [SOC 2 Type 2 certification](/workflowai-cloud/compliance). # Pricing ## Pricing WorkflowAI Cloud uses a pay-as-you-go infrastructure, similar to Amazon Web Services. There is no fixed cost, minimum spend, annual commitment, and no need to talk to sales to get started. ### Our Price match guarantee We offer a price-match guarantee for all LLM providers: WorkflowAI Cloud **charges the same per token price** as using providers directly. Currently, we support models from OpenAI, Anthropic, Google, Mistral, DeepSeek, Grok, and Llama (provided by [FireworksAI](https://fireworks.ai/)). If you have credits with Amazon, Google, or Azure, you can also continue to use them via WorkflowAI Cloud, [by providing your own API keys](/features/deployments#using-your-own-ai-providers-api). ### What does WorkflowAI Cloud charge for? WorkflowAI Cloud **only** charges for: * the tokens generated by your AI features (per token generated) * the tools used by your AI features (per tool used) We **do not** charge for: * data storage * quantity of AI features * users in your organization * bandwidth or CPU usage ### Then how does WorkflowAI Cloud make money? We make our margin by buying LLM tokens at bulk discount, and then reselling them to you at the standard public price. To break this down further, let’s look at what actually drives the cost of inference:\ The cost of inference is mostly GPU and electricity, not tokens. When you buy tokens from an LLM provider you are effectively paying for the electricity and GPU cost. But paying for GPU usage is most efficient if you can utilize the GPU at maximum capacity, all the time. That’s hard to do on your own, but possible for WorkflowAI because we’re pooling demand from our users. By taking this approach of pooling demand, WorkflowAI can keep your LLM costs low while still giving you access to a wide range of providers. ## Stripe Fulfillment Policy **Effective Date:** June 23, 2025 WorkflowAI provides instant, digital delivery of API credits through our web-based platform. This policy outlines how we handle delivery, cancellations, and refund requests for these credits. ### Delivery Method API credits are delivered digitally and automatically via your account dashboard. Upon successful payment or auto-reload (if enabled on your account) once your usage hits a pre-set threshold, credits will be added to your account. ### Delivery Timeframe Credits are typically applied within seconds of payment confirmation. In rare cases (e.g., payment processing delays), delivery may take up to 15 minutes. ### Order Confirmation Once payment has been successfully processed, you will receive an on-screen confirmation once credits have been applied. You can view your current credit balance in your account. ### Shipping No physical goods are shipped. All API credits should be purchased online via your registered account. ### Issues with Delivery If you do not see your credits after successful payment: * Log out and back into your account to refresh. * If the issue persists, contact support @ ### Cancellation * You may disable auto-reload of credits at any time through your account dashboard. * Cancellations apply only to future charges and do not affect credits that have already been delivered. * To avoid auto-reload billing, cancellations must be completed before your remaining balance reaches the auto-reload threshold. ### Refund API credit purchases are non-refundable, except in limited cases: #### Eligible for refund: * Duplicate charges * System or billing errors * Verified service outages affecting credit usability Refund requests must be submitted within 7 days of the original charge by emailing . Please include your email on file and a description of the issue. #### Refund Method: Approved refunds will be issued to the original card payment on file. #### Not Eligible for Refund: * Partially used or fully used API credits * Unused credits after account closure * Charges incurred prior to cancellation We reserve the right to deny refund requests that do not meet the criteria above or show signs of abuse. # Reliability ### Reliability [![Better Stack Badge](https://uptime.betterstack.com/status-badges/v2/monitor/1cuxx.svg)](https://status.workflowai.com) Our goal with WorkflowAI Cloud is to provide a 100% uptime on our API endpoint that is used for running an AI agent. We've designed our architecture to be resilient in a few ways: * at the AI provider level, we implemented a fallback mechanism that brings redundacy. For example, if OpenAI API is down, WorkflowAI will automatically switch to Azure OpenAI API. * at the API level, we run our inference API `run.workflowai.com` in a separate container, isolated from the rest of our other API endpoints. * at the database level, we use a multi-region database to ensure that your data is always available. * at the datacenter level, we bring redundacy by running our API in multiple independant regions. ### 1. AI provider fallback WorkflowAI continuously monitors the health and performance of all integrated AI providers. When a provider experiences downtime or degraded performance, our system automatically switches to a healthy alternative provider without any manual intervention. For example, all OpenAI models are also available through Azure OpenAI Service. If the OpenAI API becomes unavailable, WorkflowAI will automatically failover to Azure OpenAI within one second. This seamless transition ensures your agent runs continue without interruption, and you don't need to make any changes to your code. This intelligent routing between providers happens behind the scenes, maintaining consistent response times and reliability for your applications even during provider outages. ### 2. Database redundacy We use MongoDB Atlas for our primary database infrastructure, which ensures high availability through a distributed architecture with a [99.995% SLA](https://www.mongodb.com/cloud/atlas/reliability). Our database deployment includes 7 replicas across 3 Azure regions: * 3 replicas in East US2 * 2 replicas in Iowa * 2 replicas in California These replicas automatically synchronize data between them, ensuring that if one database instance or even an entire region fails, the others can immediately take over without data loss. MongoDB Atlas also offers automatic failover capabilities, where if the primary node becomes unavailable, a secondary replica is automatically promoted to primary, typically within seconds. This multi-region architecture ensures continuous database operations even during regional outages, maintenance windows, or unexpected infrastructure issues. {% hint style="info" %} For storing run history and analytics data, we use Clickhouse, which excels at handling large volumes of data efficiently. It's important to note that while Clickhouse powers our analytics and observability features, it's not required for the core agent execution functionality. The process that stores run history is completely isolated from the critical run path, ensuring that your agents will continue to run normally even if the Clickhouse database experiences temporary unavailability. {% endhint %} ### 3. Datacenter redundancy We use [Azure Front Door](https://azure.microsoft.com/en-us/products/frontdoor) as our global load balancer to ensure high availability across multiple regions. Our infrastructure is deployed in both East US and Central US datacenters, providing geographic redundancy. Azure Front Door continuously monitors the health of our backend services in each region. If one of our datacenters experiences an outage or performance degradation, Azure Front Door automatically redirects traffic to the healthy region within approximately 30 seconds. This intelligent routing happens without any manual intervention, ensuring minimal disruption to your API calls. This multi-region architecture allows us to maintain high availability even during regional cloud provider outages, helping us achieve our goal of 100% uptime for the WorkflowAI API. {% hint style="info" %} If you have any question about our architecture, please [contact us](mailto:team@workflowai.support). {% endhint %} # Compliance ## No training on your data, ever. We have contractual agreements with our AI subprocessors that prohibit the use of customer data to train their models. ## SOC2 (Type1) Compliance > Report is available [here](https://workflowai.blob.core.windows.net/workflowai-public/soc2.pdf). WorkflowAI Cloud is SOC2 Type1 compliant, ensuring that our platform meets rigorous security and compliance standards. This certification verifies the design and implementation of our security controls at a specific point in time, covering the following principles: * **Security:** Protecting against unauthorized access (both physical and logical). * **Availability:** Ensuring that the system is available for operation and use as committed. * **Processing Integrity:** Guaranteeing that system processing is complete, valid, accurate, timely, and authorized. * **Confidentiality:** Maintaining the confidentiality of information as committed or agreed. * **Privacy:** Protecting personal information according to the commitments in the privacy notice. Achieving SOC2 Type1 compliance demonstrates our commitment to maintaining the highest standards of security and operational excellence, providing our users with confidence in the integrity and reliability of our services. ## Data Hosting and Processing All customer data processed through WorkflowAI Cloud workflows is hosted and processed within the United States. ## Data Processing Agreement (DPA) We offer a Data Processing Agreement (DPA) to customers who require one for GDPR or other compliance purposes. Please contact to request a copy. ## GDPR & Frequently Asked Questions ### Where is my data hosted and processed? Your data is hosted and processed in data centers located in the United States. ### Is data transferred outside the EU/EEA? What safeguards are in place? Yes, since data is processed in the US, it is transferred outside the European Union / European Economic Area. We provide a Data Processing Agreement (DPA) which outlines the contractual commitments between WorkflowAI and our customers. Customers are responsible for ensuring they have a lawful basis for transferring personal data to the US when using our services. Please contact for our DPA. ### Do you offer a Data Processing Agreement (DPA)? Yes, please contact to request our DPA. ### Is the data sent to AI models retained? For how long? Can it be deleted? We have contractual agreements with our AI subprocessors that prohibit them from using your data to train their models. However, some subprocessors may temporarily retain data for operational purposes according to their own policies. Retention periods vary by subprocessor and specific operational needs. WorkflowAI does not control these subprocessor retention periods directly, and we cannot force deletion from their systems outside of our contractual 'no training' guarantees. ### Which AI subprocessors do you use? We utilize various leading AI model providers. Our contractual agreements with all subprocessors ensure your data is not used for training purposes. For specific details on subprocessors relevant to your usage, please refer to our DPA or contact support. ### Can I process sensitive personal data (e.g., identity documents) on WorkflowAI? WorkflowAI does not have specific technical restrictions preventing the processing of various data types. However, we currently do not have a formal policy explicitly governing the processing of sensitive personal data (such as identity documents, health information, etc.). Customers are solely responsible for ensuring their use of WorkflowAI complies with all applicable regulations, including GDPR, especially concerning sensitive data categories which often require heightened safeguards and explicit consent. We advise exercising caution and recommend discussing specific use cases involving sensitive data with your legal counsel and contacting us if you have further questions. ## Self-Hosting for Maximum Control For organizations with particularly strict compliance, data residency, or security requirements, we recommend considering the self-hosted version of WorkflowAI. Self-hosting provides maximum control over your data environment. You can find information on setting up self-hosting in our GitHub repository: . Please don't hesitate to contact our sales team via email for more information and support on self-hosting options. > If you have specific compliance questions and requirements (HIPPA, GDPR, data retention, etc) not covered here, please contact us at . # Get started {% hint style="info" %} For engineers. {% endhint %} WorkflowAI Python SDK is a library that allows you to programmatically create and run agents in Python, while being able to use the full power of the WorkflowAI platform. ## Install the SDK {% hint style="info" %} `workflowai` requires Python >= 3.9. {% endhint %} [![PyPI version](https://img.shields.io/pypi/v/workflowai.svg)](https://pypi.org/project/workflowai/) ```sh pip install workflowai ``` {% hint style="info" %} [Star the repository on Github](https://github.com/WorkflowAI/python-sdk) to get notified when new models are added. {% endhint %} ## API Key Get your API key from your [WorkflowAI Cloud dashboard](https://workflowai.com/organization/settings/api-keys) or from your self-hosted WorkflowAI dashboard. Set the `WORKFLOWAI_API_KEY` environment variable. ## Initialize the SDK ```python import os import workflowai workflowai.init( # This initialization is optional when using default settings api_key=os.environ.get("WORKFLOWAI_API_KEY"), # This is the default and can be omitted url="https://run.workflowai.com", # This is the default and can be omitted ) ``` {% hint style="success" %} `run.workflowai.com` is our [globally distributed, highly available endpoint](/workflowai-cloud/reliability) {% endhint %} {% hint style="info" %} You can also set the `WORKFLOWAI_API_URL` environment variable to point to your self-hosted WorkflowAI. {% endhint %} ## Write your first agent An agent is in essence an async function with the added constraints that: * it has a single argument that is a Pydantic model, which is the input to the agent * it has a single return value that is a Pydantic model, which is the output of the agent * it is decorated with the `@workflowai.agent()` decorator {% hint style="info" %} [Pydantic](https://docs.pydantic.dev/latest/) is a very popular and powerful library for data validation and parsing. {% endhint %} The following agent, given a city, returns the country, capital, and a fun fact about the city. ```python import workflowai from pydantic import BaseModel from workflowai import Model class CityInput(BaseModel): city: str class CapitalOutput(BaseModel): country: str capital: str fun_fact: str @workflowai.agent() async def get_capital_info(city_input: CityInput) -> CapitalOutput: ... ``` {% tabs %} {% tab title="OpenAI" %} ```python output = await get_capital_info.run( CityInput(city="New York"), model=Model.GPT_4_LATEST ) print(output) # { # "country": "United States", # "capital": "Washington, D.C.", # "fun_fact": "New York City is known as 'The Big Apple' and is famous for its cultural diversity and iconic landmarks like Times Square and Central Park." # } # ================================================== # Cost: $ 0.00091 # Latency: 1.65s ``` {% endtab %} {% tab title="Anthropic" %} ```python output = await get_capital_info.run( CityInput(city="New York"), model=Model.CLAUDE_3_5_SONNET_LATEST ) print(output) # Output: # ================================================== # { # "country": "United States", # "capital": "Washington, D.C.", # "fun_fact": "New York City's Federal Reserve Bank has the largest gold storage in the world, containing approximately 7,000 tons of gold bullion stored 80 feet below street level." # } # ================================================== # Cost: $ 0.001755 # Latency: 2.43s ``` {% endtab %} {% tab title="Gemini" %} ```python output = await get_capital_info.run( CityInput(city="New York"), model=Model.GEMINI_2_0_FLASH_LATEST ) print(output) # Output: # ================================================== # { # "country": "United States of America", # "capital": "Washington, D.C.", # "fun_fact": "New York City is home to over 8 million people and over 800 languages are spoken in New York City, making it the most linguistically diverse city in the world." # } # ================================================== # Cost: $ 0.00005 # Latency: 1.26s ``` {% endtab %} {% tab title="OpenAI (stream)" %} ```python # use `.stream()` to stream the output async for chunk in get_capital_info.stream( CityInput(city="New York"), model=Model.GPT_4O_MINI_LATEST ): print(chunk) # { # "country": "United" # } # { # "country": "United States" # } # { # "country": "United States" # "capital": "Washington" # } # streaming continues... # ... ``` {% endtab %} {% endtabs %} {% hint style="success" %} You have created your first agent! Congratulations. {% endhint %} Agents created by the SDK are also available in the [Playground](https://workflowai.com/docs/agents/get-capital-info/1). [![Playground](https://2418444523-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FW4ng0K5LfFjYqHYuPgNh%2Fuploads%2Fgit-blob-d2cd8c45d767a51f39d8edd56285d652fd0b6ec4%2Fdocs-capital-info.png?alt=media)](https://workflowai.com/docs/agents/get-capital-info/1) Runs are automatically logged as well from the [Runs](https://workflowai.com/docs/agents/get-capital-info/1/runs?page=0) section. [![Runs](https://2418444523-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FW4ng0K5LfFjYqHYuPgNh%2Fuploads%2Fgit-blob-2b863fdeb62788a2a1915bdc943a0b0f6d68fba5%2Fdocs-capital-info.png?alt=media)](https://workflowai.com/docs/agents/get-capital-info/1/runs?page=0) ## Next steps Let's go through in more detail [how to setup an agent](/python-sdk/agent). # @workflowai.agent {% hint style="info" %} [Star the repository on Github](https://github.com/WorkflowAI/python-sdk) to get notified when new models are added. {% endhint %} ## Introduction WorkflowAI takes a different approach to LLMs that simplifies development while providing more structure and reliability. ### Traditional LLM Approach vs. WorkflowAI Let's compare approaches with a practical example: extracting positive and negative aspects from customer reviews. #### Traditional Approach With traditional LLM frameworks, you might create a prompt like this: ``` Based on the following reviews: {{ reviews }} Identify what are the positive and negative aspects. Make sure you return the output in JSON format: { "positive_aspects": [string], "negative_aspects": [string] } ``` This prompt combines several elements: | Component | Description | | ------------- | ----------------------------------------------------------- | | Instructions | "Identify what are the positive and negative aspects." | | Variables | `{{ reviews }}` - Data to be processed | | Output format | JSON structure with arrays of positive and negative aspects | Despite explicitly requesting JSON output, there's no guarantee the model will comply. The LLM might return malformed JSON, skip the format entirely, or include incorrect fields—requiring you to write additional validation code. #### WorkflowAI Approach The same task in WorkflowAI becomes more structured and type-safe: ```python class FeedbackInput(BaseModel): reviews: list[str] class FeedbackOutput(BaseModel): positive_aspects: list[str] negative_aspects: list[str] @workflowai.agent(id="feedback") async def feedback(input: FeedbackInput) -> FeedbackOutput: """ Identify what are the positive and negative aspects. """ ... run = await feedback.run( FeedbackInput(reviews=["..."]), model=Model.GPT_4O_LATEST ) ``` WorkflowAI guarantees your output will match the defined schema by validating responses and automatically handling invalid data. No more worrying about malformed JSON or writing extensive error handling code. WorkflowAI generates optimal prompts by combining your Pydantic models, docstring instructions, and any additional context. Benefits include automatic type validation, cleaner code architecture, and consistently reliable outputs. Let's explore how this works by breaking down the different parts of an agent: 1. [Schema](#schema-input-output) (input, output) 2. [Instructions](#instructions) 3. [Model](#model) Optionally, an agent can also have tools, which will be explained in the [Tools](/python-sdk/tools) section. ## Schema (input, output) The schema has two structured parts: | | | | ---------- | ---------------------------------------------------------- | | **Input** | Defines the variables that the agent will receive as input | | **Output** | Defines the variables that the agent will return as output | The input and output are defined using [Pydantic](https://docs.pydantic.dev/latest/) models. A very simple example of a schema is the following, where the agent receives a question as input and returns an answer as output. ```python from pydantic import BaseModel class Input(BaseModel): question: str class Output(BaseModel): answer: str ``` {% hint style="success" %} Read more about why schemas are a good idea in the [Schemas](/concepts/schemas#why-are-schemas-a-good-idea) section. {% endhint %} {% hint style="info" %} Find more examples of schemas in the [Schemas](/python-sdk/schemas) section. {% endhint %} ### Descriptions Adding descriptions to the input and output fields is optional, but it's a good practice to do so, as descriptions will be included in the final prompt sent to the LLM, and will help align the agent's behavior. ```python class Output(BaseModel): answer: str = Field(description="Answer with bullet points.") ``` ### Examples Another effective way to align the agent's behavior is to provide examples for **output** fields. ```python class Output(BaseModel): answer: str = Field( description="Answer with bullet points.", examples=[ "- Answer 1", "- Answer 2", "- Answer 3" ] ) ``` {% hint style="info" %} There are very little use cases for descriptions and examples in the **input** fields. The LLM will most of the time infer from the value that is passed. {% endhint %} ### Required versus optional fields In short, we recommend using default values for most output fields. Pydantic is by default rather strict on model validation. If there is no default value, the field must be provided.\ Although the fact that a field is required is passed to the model, the generation can sometimes omit null or empty values. ## Instructions Instructions are helpful for the agent to understand the task it needs to perform. Use docstring to add instructions to the agent. ```python @workflowai.agent(id="answer-question") async def answer_question(input: Input) -> Output: """ You are an expert in history. Answer the question with attention to detail and historical accuracy. """ ... ``` Instructions are automatically passed to the LLM via the system prompt. ``` system_prompt = """You are an expert in history. Answer the question with attention to detail and historical accuracy.""" ``` ### Variables in instructions You can customize your agent's instructions using [Jinja2](https://jinja.palletsprojects.com/) template variables in the docstring. These variables are automatically filled with values from your input model's fields, giving you precise control over the final prompt. ```python class Input(BaseModel): question: str word_count: int class Output(BaseModel): answer: str @workflowai.agent(id="answer-question-with-word-count", model=Model.CLAUDE_3_5_HAIKU_LATEST) async def answer_question(input: Input) -> Output: """ The answer should be less than {{ word_count }} words. Answer the following question: {{ question }} """ ... # Run the agent run = await answer_question.run( Input( question="What is artificial intelligence?", word_count=5 ) ) # View prompt # https://workflowai.com/docs/agents/answer-question-with-word-count/runs/019509ed-017e-7059-4c25-6137ebdb7dcd # System prompt: # The answer should be less than 5 words. Answer the following question: What is artificial intelligence? # { "answer": "Smart computer systems learning" } ```
Example: Code Review Agent ```python class CodeReviewInput(BaseModel): language: str = Field(description="Programming language of the code") style_guide: str = Field(description="Style guide to follow") is_production: bool = Field(description="Whether this is a production review") focus_areas: list[str] = Field(description="Areas to focus on during review", default_factory=list) class CodeReviewOutput(BaseModel): """Output from a code review.""" issues: list[str] = Field( default_factory=list, description="List of identified issues or suggestions for improvement" ) compliments: list[str] = Field( default_factory=list, description="List of positive aspects and good practices found in the code" ) summary: str = Field( description="A brief summary of the code review findings" ) @workflowai.agent(id="code-review") async def review_code(review_input: CodeReviewInput) -> CodeReviewOutput: """ You are a code reviewer for {{ language }} code. Please review according to the {{ style_guide }} style guide. {% if is_production %} This is a PRODUCTION review - be extra thorough and strict. {% else %} This is a development review - focus on maintainability. {% endif %} {% if focus_areas %} Key areas to focus on: {% for area in focus_areas %} {{ loop.index }}. {{ area }} {% endfor %} {% endif %} Code to review: {{ code }} """ ... ```
We recommend using CursorAI, Claude or ChatGPT to help generate the Jinja2 template. The template uses [Jinja2](https://jinja.palletsprojects.com/) syntax and supports common templating features including: * Variable substitution: `{{ variable }}` * Conditionals: `{% if condition %}...{% endif %}` * Loops: `{% for item in items %}...{% endfor %}` * Loop indices: `{{ loop.index }}` See the [Jinja2 documentation](https://jinja.palletsprojects.com/) for the full template syntax and capabilities. ### Temperature The temperature is a parameter that controls the randomness of the output. It is a float between 0 and 1. The default temperature is 0. ```python run = await answer_question.run( Input(question="What is the history of Paris?"), temperature=0.5 ) ``` ## Model The model is the LLM that will be used to generate the output. WorkflowAI offers a unified interface for all the models it supports from OpenAI, Anthropic, Google, and more. Simply pass the model you want to use to the `model` parameter. {% hint style="info" %} The [list of models supported by WorkflowAI is available here](https://github.com/WorkflowAI/python-sdk/blob/main/workflowai/core/domain/model.py), but you can also see the list of models from the playground, for a more user-friendly experience. {% endhint %} Set the model in the `@agent` decorator. ```python import workflowai from workflowai import Model @workflowai.agent(id="answer-question", model=Model.GPT_4O_LATEST) async def answer_question(input: Input) -> Output: ... ``` {% hint style="info" %} When a model is retired or deprecated, WorkflowAI automatically upgrades it to the latest compatible version with equivalent or better pricing. This ensures your agents continue working seamlessly without any code changes needed on your end. {% endhint %} ### Supported models When building an agent that uses images, or audio, you need to use a model that supports multimodality. Use the `list_models()` function to get the list of models and check if they support your use case by checking the `is_not_supported_reason` field. ```python class AudioInput(BaseModel): audio: Audio = Field() class AudioOutput(BaseModel): transcription: str = Field() @agent(id="audio-transcription") async def audio_transcription(input: AudioInput) -> AudioOutput: """ Transcribe the audio file. """ ... models = await audio_transcription.list_models() for model in models: if model.is_not_supported_reason is None: print(f"{model.id} supports audio transcription") else: print(f"{model.id} does not support audio transcription: {model.is_not_supported_reason}") # ... ``` {% hint style="info" %} The `list_models()` function is a powerful way to programmatically discover which models are compatible with your agent's requirements. This is especially important for multimodal agents that handle images or audio, as not all models support these capabilities. You can use this information to dynamically select the most appropriate model at runtime or to provide fallback options. {% endhint %} ## Running the agent {% hint style="warning" %} Before you run the agent, make sure you have [setup the WorkflowAI client](/python-sdk/get-started#api-key). {% endhint %} To run the agent, simply call the `run` function with an input. ```python run = await answer_question.run(Input(question="What is the history of Paris?")) print(run) # Output: # ================================================== # { # "answer": "- Paris, the capital of France, has a history that dates back to ancient times, originally settled by the Parisii, a Celtic tribe, around 250 BC.\n- During the Roman era, it was known as Lutetia and became a significant city in the Roman province of Gaul.\n- In the Middle Ages, Paris grew as a center of learning and culture, with the establishment of the University of Paris in the 12th century.\n- The city played a pivotal role during the French Revolution in the late 18th century, becoming a symbol of revolutionary ideals.\n- In the 19th century, Paris underwent major transformations under Baron Haussmann, who modernized the city's infrastructure and architecture.\n- Paris was occupied during World War II but was liberated in 1944, marking a significant moment in its modern history.\n- Today, Paris is renowned for its cultural heritage, iconic landmarks like the Eiffel Tower and Notre-Dame Cathedral, and its influence in art, fashion, and politics." # } # ================================================== # Cost: $ 0.0027 # Latency: 6.54s ``` When you call `run`, the associated agent will be created on WorkflowAI Cloud (or your self-hosted server) if it does not already exist. {% hint style="info" %} The agent id will be a slugified version of the function name unless specified explicitly using the `id` parameter, which is **recommended**. ```python @workflowai.agent(id="answer-question") async def answer_question(input: Input) -> Output: ... ``` {% endhint %} ### Override the default model You can also pass a `model` parameter to the agent function itself to specify the model you want to use, and override the default model set in the `@agent` decorator. ```python run = await answer_question.run( Input(question="What is the history of Paris?"), model=Model.CLAUDE_3_5_SONNET_LATEST ) print(run) ``` ### Cost, latency WorkflowAI automatically tracks the cost and latency of each run, and makes it available in the `run` object. ```python run = await answer_question.run(Input(question="What is the history of Paris?")) print(f"Cost: $ {run.cost_usd:.5f}") print(f"Latency: {run.duration_seconds:.2f}s") # Cost: $ 0.00745 # Latency: 8.99s ``` ### Streaming WorkflowAI also support streaming the output, using the `stream` method. The `stream` method returns an AsyncIterator, so you can use it in an async for loop. ```python async for chunk in answer_question.stream(Input(question="What is the history of Paris?")): print(chunk) # Output: # ================================================== # { # "answer": "-" # } # ================================================== # Output: # ================================================== # { # "answer": "- Founde" # } # ================================================== # Output: # ================================================== # { # "answer": "- Founded aroun" # } # ================================================== # Output: # ================================================== # { # "answer": "- Founded around 250" # } # ================================================== # Output: # ================================================== # { # "answer": "- Founded around 250 BCE" # } # ================================================== # ... ``` {% hint style="info" %} Even when using streaming, partial outputs are returned as valid output schemas. {% endhint %} ### View the prompt To access the exact prompt sent by WorkflowAI to any AI provider, and the raw response as well, you can use `fetch_completions` on a run object. For example: ````python # Fetch the raw completion from the LLM run = await answer_question.run(Input(question="What is the history of Paris?")) # Get completion details completions = await run.fetch_completions() for completion in completions: completion_json = completion.model_dump_json(indent=2) print(completion_json) # Output: # { # "messages": [ # { # "role": "system", # "content": "\nYou are an expert in history.\nAnswer the question with attention to detail and historical accuracy.\n\n\nInput will be provided in the user message using a JSON following the schema:\n```json\n{\n \"properties\": {\n \"question\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"question\"\n ],\n \"type\": \"object\"\n}\n```" # }, # { # "role": "user", # "content": "Input is:\n```json\n{\n \"question\": \"What is the history of Paris?\"\n}\n```" # } # ], # "response": "{\"answer\":\"- Paris, the capital of France, has a history that dates back to ancient times, originally settled by the Parisii, a Celtic tribe, around 250 BC...\"}", # "usage": { # "completion_token_count": 177, # "completion_cost_usd": 0.00177, # "reasoning_token_count": 0, # "prompt_token_count": 210, # "prompt_token_count_cached": 0, # "prompt_cost_usd": 0.0005250000000000001, # "prompt_audio_token_count": 0, # "prompt_audio_duration_seconds": 0.0, # "prompt_image_count": 0, # "model_context_window_size": 128000 # } # } ```` {% hint style="info" %} The `fetch_completions` method is particularly useful for debugging, understanding token usage, and auditing the exact interactions with the underlying AI models. This can help you optimize prompts, analyze costs, and ensure the model is receiving the expected instructions. {% endhint %} ### Error handling Read more about error handling in the [Errors](/python-sdk/errors) section. ### Cache To save money and improve latency, WorkflowAI supports caching. By default, the cache settings is `auto`, meaning that agent runs are cached when the temperature is 0\ (the default temperature value) and no tools are used. Which means that, when running the same agent (without tools) twice with the **exact** same input, the exact same output is returned and the underlying model is not called a second time. The cache usage string literal is defined in [cache\_usage.py](https://github.com/WorkflowAI/workflowai-py/blob/main/workflowai/core/domain/cache_usage.py) file. There are 3 possible values: * `auto`: (default) Use cached results only when temperature is 0, and no tools are used * `always`: Always use cached results if available, regardless of model temperature * `never`: Never use cached results, always execute a new run ```python # Never use cache run = agent.run(input, use_cache='never') # Always use cache run = agent.run(input, use_cache='always') # Auto (default): use cache when temperature is 0 and no tools are used run = agent.run(input) ``` ## Reply to a run For some use-cases (for example, chatbots), you want to reply to a previously created run to maintain conversation history. Use the `reply` method from the `Run` object. For example, a simple travel chatbot agent can be created as follows: ```python class ChatbotInput(BaseModel): user_message: str class Recommendation(BaseModel): name: str address: str class ChatbotOutput(BaseModel): assistant_message: str # You can add structured output to the assistant reply recommendations: list[Recommendation] @workflowai.agent(id="travel-assistant", model=Model.GPT_4O_LATEST) async def chat(input: ChatbotInput) -> ChatbotOutput: """ A helpful travel assistant that can provide recommendations and answer questions about destinations. """ ... # Initial question from user run = await chat.run(ChatbotInput(user_message="I'm planning a trip to Paris. What are the must-see attractions?")) # Output: # ================================================== # { # "assistant_message": "Paris is a city rich in history, culture, and beauty. Here are some must-see attractions to include in your itinerary.", # "recommendations": [ # { # "name": "Eiffel Tower", # "address": "Champ de Mars, 5 Avenue Anatole France, 75007 Paris, France" # }, # ... # ] # } ``` When using `run.reply`, WorkflowAI will automatically keep the conversation history. {% hint style="warning" %} Note that the output schema of the reply will use the same output schema as the original run. {% endhint %} ```python # Note that the follow-up question does not mention Paris because the conversation history is automatically kept. reply_run = await run.reply(user_message="When is the best time of year to visit?") print(reply_run) # Output: # Note that the output schema include a `recommendations` field, because the output schema of the original run includes a `recommendations` field. # ================================================== # { # "assistant_message": "The best time to visit Paris is during the spring (April to June) and fall (September to November) seasons. During these months, the weather is generally mild and pleasant, and the city is less crowded compared to the peak summer months. Spring offers blooming flowers and vibrant parks, while fall provides a charming atmosphere with colorful foliage. Additionally, these periods often feature cultural events and festivals, enhancing the overall experience of your visit.", # "recommendations": [] # } # ================================================== # Cost: $ 0.00206 # Latency: 2.08s ``` You can continue to reply to the run as many times as you want. Another use-case for `run.reply` is to ask a follow-up question, or ask the LLM to double-check its previous answer. ```python # Double-check the answer confirmation_run = await run.reply( user_message="Are you sure?" ) ``` ## Using multiple clients You might want to avoid using the shared client, for example if you are using multiple API keys or accounts.\ It is possible to achieve this by manually creating client instances ```python from workflowai import WorkflowAI client = WorkflowAI( url=..., api_key=..., ) # Use the client to create and run agents @client.agent() def my_agent(agent_input: Input) -> Output: ... ``` ## Field properties Pydantic allows a variety of other validation criteria for fields: minimum, maximum, pattern, etc.\ This additional criteria are included the JSON Schema that is sent to WorkflowAI, and are sent to the model. ```python class Input(BaseModel): name: str = Field(min_length=3, max_length=10) age: int = Field(ge=18, le=100) email: str = Field(pattern=r"^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$") ``` These arguments can be used to stir the model in the right direction. The caveat is have a\ validation that is too strict can lead to invalid generations. In case of an invalid generation: * WorkflowAI retries the inference once by providing the model with the invalid output and the validation error * if the model still fails to generate a valid output, the run will fail with an `InvalidGenerationError`.\ the partial output is available in the `partial_output` attribute of the `InvalidGenerationError` # Schemas List of common schemas. {% hint style="info" %} Read first about schemas in the [`@workflowai.agent`](/python-sdk/agent#schema-input-output) section. {% endhint %} ## Chatbot A chatbot is represented by a back-and-forth conversation between a user and an assistant. ```python class ChatbotInput(BaseModel): user_message: str class ChatbotOutput(BaseModel): assistant_message: str ``` {% hint style="info" %} Read more about chatbots in the [`@workflowai.agent`](/python-sdk/agent#reply-to-a-run) `reply` method section. {% endhint %} ## PII Extraction PII extraction is the process of identifying and extracting specific entities from a text. ```python class PIIType(str, Enum): """Categories of Personal Identifiable Information.""" NAME = "NAME" # Full names, first names, last names EMAIL = "EMAIL" # Email addresses PHONE = "PHONE" # Phone numbers, fax numbers ADDRESS = "ADDRESS" # Physical addresses, postal codes SSN = "SSN" # Social Security Numbers, National IDs DOB = "DOB" # Date of birth, age FINANCIAL = "FINANCIAL" # Credit card numbers, bank accounts LICENSE = "LICENSE" # Driver's license, professional licenses URL = "URL" # Personal URLs, social media profiles OTHER = "OTHER" # Other types of PII not covered above class PIIExtraction(BaseModel): """Represents an extracted piece of PII with its type.""" text: str = Field(description="The extracted PII text") type: PIIType = Field(description="The category of PII") start_index: int = Field(description="Starting position in the original text") end_index: int = Field(description="Ending position in the original text") class PIIInput(BaseModel): """Input model for PII extraction.""" text: str = Field(description="The text to analyze for PII") class PIIOutput(BaseModel): """Output model containing redacted text and extracted PII.""" redacted_text: str = Field( description="The original text with all PII replaced by [REDACTED]", examples=[ "Hi, I'm [REDACTED]. You can reach me at [REDACTED] or call [REDACTED]. " "My SSN is [REDACTED] and I live at [REDACTED].", ], ) extracted_pii: list[PIIExtraction] = Field( description="List of extracted PII items with their types and positions", examples=[ [ {"text": "John Doe", "type": "NAME", "start_index": 8, "end_index": 16}, {"text": "john.doe@email.com", "type": "EMAIL", "start_index": 30, "end_index": 47}, {"text": "555-0123", "type": "PHONE", "start_index": 57, "end_index": 65}, ], ], ) ``` ## Extract positive and negative points from transcript ```python class FeedbackInput(BaseModel): """Input for analyzing a customer feedback call.""" transcript: str = Field(description="The full transcript of the customer feedback call.") call_date: date = Field(description="The date when the call took place.") # Model representing a single feedback point with supporting evidence class FeedbackPoint(BaseModel): """A specific feedback point with its supporting quote.""" point: str = Field(description="The main point or insight from the feedback.") quote: str = Field(description="The exact quote from the transcript supporting this point.") timestamp: str = Field(description="The timestamp or context of when this was mentioned in the call.") # Model representing the structured analysis of the customer feedback call class FeedbackOutput(BaseModel): """Structured analysis of the customer feedback call.""" positive_points: list[FeedbackPoint] = Field( default_factory=list, description="List of positive feedback points, each with a supporting quote." ) negative_points: list[FeedbackPoint] = Field( default_factory=list, description="List of negative feedback points, each with a supporting quote." ) ``` ## Image {% hint style="info" %} Read about image schemas in the [Multimodality](/python-sdk/multimodality#images) section. {% endhint %} ```python class ImageInput(BaseModel): image: Image = Field(description="The image to analyze") class ImageOutput(BaseModel): city: str = Field(default="", description="Name of the city shown in the image") country: str = Field(default="", description="Name of the country where the city is located") confidence: Optional[float] = Field( default=None, description="Confidence level in the identification (0-1)", ) ``` ## Audio {% hint style="info" %} Read about audio schemas in the [Multimodality](/python-sdk/multimodality#audio) section. {% endhint %} ```python class AudioInput(BaseModel): """Input containing the audio file to analyze.""" audio: Audio = Field( description="The audio recording to analyze for spam/robocall detection", ) class SpamIndicator(BaseModel): """A specific indicator that suggests the call might be spam.""" description: str = Field( description="Description of the spam indicator found in the audio", examples=[ "Uses urgency to pressure the listener", "Mentions winning a prize without entering a contest", "Automated/robotic voice detected", ], ) quote: str = Field( description="The exact quote or timestamp where this indicator appears", examples=[ "'You must act now before it's too late'", "'You've been selected as our prize winner'", "0:05-0:15 - Synthetic voice pattern detected", ], ) class AudioClassification(BaseModel): """Output containing the spam classification results.""" is_spam: bool = Field( description="Whether the audio is classified as spam/robocall", ) confidence_score: float = Field( description="Confidence score for the classification (0.0 to 1.0)", ge=0.0, le=1.0, ) spam_indicators: list[SpamIndicator] = Field( default_factory=list, description="List of specific indicators that suggest this is spam", ) reasoning: str = Field( description="Detailed explanation of why this was classified as spam or legitimate", ) ``` # Versions WorkflowAI agents are versioned automatically. To show how versions work, let's create a new agent that can triage a customer question into different categories. ```python from typing import Literal from pydantic import BaseModel import workflowai class Input(BaseModel): question: str class Output(BaseModel): category: Literal["billing", "technical", "account", "other"] @workflowai.agent(id="triage-agent") async def triage_question(input: Input) -> Output: """ Triage a customer question into different categories. """ ... await triage_question.run(Input(question="How do I change my billing information?")) ``` {% hint style="info" %} You can test this agent yourself on [WorkflowAI](https://workflowai.com/docs/agents/triage-agent/1). {% endhint %} Running this agent for the first time will automatically create a new version of the agent on WorkflowAI. ![Versions](https://2418444523-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FW4ng0K5LfFjYqHYuPgNh%2Fuploads%2Fgit-blob-3a5b7a52ab7b3503794473b7f9ffa9cf75038c19%2Fversion-1.png?alt=media) A version is a specific **configuration** of an agent. WorkflowAI defines two types of (agent) versions: | Version Type | Example | Description | | ------------------ | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------ | | **Major** Versions | 1, 2, 3, ... | A major version represents a specific configuration of a agent, including its instructions, temperature, descriptions/examples, and tools. | | **Minor** Versions | 1.1, 1.2, 1.3, ... | A minor version represents a major version **associated with a specific model** (e.g., OpenAI's GPT-4o-mini). | Now let's create another version of the agent, but this time we'll use a different model. ```python @workflowai.agent( id="triage-agent", model=Model.CLAUDE_3_5_HAIKU_LATEST ) async def triage_question(input: Input) -> Output: """ Triage a customer question into different categories. """ ... await triage_question.run(Input(question="How do I change my billing information?")) ``` This will create a new minor version of the agent associated with the `CLAUDE_3_5_HAIKU_LATEST` model. ![Versions that have the same parameters are grouped together](https://2418444523-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FW4ng0K5LfFjYqHYuPgNh%2Fuploads%2Fgit-blob-0f8ef0e272ae5eea8a6f7af45e7c2e42d7ec476d%2Fversion-1.2.png?alt=media) ## Major Versions Major versions are created when you change the instructions, temperature, descriptions/examples, or tools of an agent. For example, let's change the instructions of the agent. ```python async def triage_question(input: Input) -> Output: """ Triage a customer question into different categories. Categories: - billing: Questions about payments, invoices, pricing, or subscription changes - technical: Questions about API usage, SDK implementation, or technical issues - account: Questions about account access, settings, or profile management - other: Questions that don't fit into the above categories """ ... ``` {% hint style="info" %} Changelog between major versions will be generated automatically. {% endhint %} ![Version 2 with new instructions](https://2418444523-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FW4ng0K5LfFjYqHYuPgNh%2Fuploads%2Fgit-blob-2a5b20354acd4ad28a5d3631852eb459a79cfde5%2Fversion-2.png?alt=media) ## Versions from code, version id, or a deployment WorkflowAI allows you to refer to a version of an agent from your code, a minor version id, or a deployment. ### Versions from code Setting a docstring or a model in the `@workflowai.agent` decorator signals the client that the agent parameters are fixed and configured via code. ### Versions from version id Since WorkflowAI automatically saves all versions, you can refer to a minor version by its id. ```python # this agent will use the version 2.1 @workflowai.agent(id="triage-agent", version="2.1") ``` {% hint style="info" %} You can also go to the [Code](https://workflowai.com/docs/agents/triage-agent/1/code?selectedLanguage=Python) section on WorkflowAI to view the generated code for a specific version. {% endhint %} ### Versions from a deployment {% hint style="info" %} To learn more about deployments, read the [Deployments](/features/deployments) section first. {% endhint %} Deployments allow you to refer to a version of an agent's parameters from your code that's managed from WorkflowAI dashboard, allowing you to update the agent's parameters without changing your code. ```python # production @workflowai.agent(id="triage-agent", deployment="production") # or simply @workflowai.agent() # development @workflowai.agent(id="triage-agent", deployment="development") # staging @workflowai.agent(id="triage-agent", deployment="staging") ``` # Deployments ## What are deployments? Deploy specific versions of an agent with ease, allowing for updates to prompts and models **without any code changes**. ## Why are deployments useful? * ✅ update to a new model or prompt without asking your engineering team. * ✅ save cost by updating to a more recent, cheaper model, without changing your code. * ✅ improve the quality of your tasks outputs by adjusting the prompt, in real-time, based on users' feedback. * ✅ use different versions of a task in different environments (development, staging, production). ## How to deploy a version? 1. Go to [workflowai.com](https://workflowai.com) and login. 2. Go to **Deployments** section from the menu. 3. Pick the environment you want to deploy to, either: production, staging, or development. 4. Tap **Deploy Version** 5. Select the version you want to deploy. 6. Tap **Deploy** After deploying a version: you will be able to reference the version you want to use by its environment in your code. Anytime you want to update the version, you can do so by going to the **Deployments** section and deploying a new version to the same environment, no code changes are required. {% embed url="" %} {% hint style="warning" %} To avoid any breaking changes: deployments are **schema specific**, not AI feature specific. This means that if you want to deploy a new version of your AI feature that is on a different schema, you will need to update the schema number in your code. This also means that you can deploy a (development, staging, production) version for each schema of an agent without the version deployed to production being affected. {% endhint %} # Multimodality Build agents that can handle multiple modalities, such as images, PDF, documents, audio, and video. ## Images Add images as input to an agent by using the `Image` class. An image can either have: * a `content`, base64 encoded data * a `url` In the following example, we'll build an agent that can identify the city and country from an image. ```python import base64 import workflowai from workflowai import Model from workflowai.fields import Image from pydantic import BaseModel, Field from typing import Optional class ImageInput(BaseModel): image: Image = Field(description="The image to analyze") class ImageOutput(BaseModel): city: str = Field(default="", description="Name of the city shown in the image") country: str = Field(default="", description="Name of the country where the city is located") confidence: Optional[float] = Field( default=None, description="Confidence level in the identification (0-1)", ) @workflowai.agent(id="city-identifier", model=Model.GEMINI_2_0_FLASH_LATEST) async def identify_city_from_image(image_input: ImageInput) -> ImageOutput: """ Analyze the provided image and identify the city and country shown in it. If the image shows a recognizable landmark or cityscape, identify the city and country. If uncertain, indicate lower confidence or leave fields empty. Focus on: - Famous landmarks - Distinctive architecture - Recognizable skylines - Cultural elements that identify the location Return empty strings if the city/country cannot be determined with reasonable confidence. """ ... # Run the agent ## From a remote image image_url = "https://images.pexels.com/photos/699466/pexels-photo-699466.jpeg" image = Image(url=image_url) agent_run = await identify_city_from_image.run(ImageInput(image=image)) print(agent_run) # Output: # ================================================== # { # "city": "Paris", # "country": "France", # "confidence": 1.0 # } # ================================================== # Cost: $ 0.00024 # Latency: 6.70s ## From a local image image_path = "path/to/image.jpg" with open(image_path, "rb") as image_file: content = base64.b64encode(image_file.read()).decode("utf-8") image = Image(content_type="image/jpeg", data=content) agent_run = await identify_city_from_image.run(ImageInput(image=image)) ``` See a more complete example in [examples/07\_image\_agent.py](https://github.com/WorkflowAI/python-sdk/blob/main/examples/07_image_agent.py). {% hint style="info" %} You can test this agent with your own images in the WorkflowAI playground. View on [WorkflowAI](https://workflowai.com/docs/agents/city-identifier/1). {% endhint %} ![Compare models](https://2418444523-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FW4ng0K5LfFjYqHYuPgNh%2Fuploads%2Fgit-blob-190fb9955f6d23e9d4d3ef6efe24fdba2ec698e9%2Fplayground.png?alt=media) {% hint style="info" %} Images generation is not supported yet. {% endhint %} ## PDF, documents Use the `PDF` class to add PDF, documents, or other files as input to an agent. In the following example, an agent is created that can answer questions based on a PDF document. {% hint style="info" %} This example uses the `GEMINI_2_0_FLASH_LATEST` model - a powerful and cost-effective multimodal model optimized for processing PDF documents. {% endhint %} ```python import workflowai from pydantic import BaseModel, Field from workflowai import Model from workflowai.fields import PDF class PDFQuestionInput(BaseModel): pdf: PDF = Field(description="The PDF document to analyze") question: str = Field(description="The question to answer about the PDF content") class PDFAnswerOutput(BaseModel): answer: str = Field(description="The answer to the question based on the PDF content") quotes: list[str] = Field(description="Relevant quotes from the PDF that support the answer") @workflowai.agent(id="pdf-answer-bot", model=Model.GEMINI_2_0_FLASH_LATEST) async def answer_pdf_question(input: PDFQuestionInput) -> PDFAnswerOutput: """ Analyze the provided PDF document and answer the given question. Provide a clear and concise answer based on the content found in the PDF. Include relevant quotes to support the answer. """ ... # Run the agent ## From a remote file pdf_url = "https://microsoft.gcs-web.com/static-files/b3eef820-6757-44ea-9f98-3963bace4837" pdf = PDF(url=pdf_url) agent_run = await answer_pdf_question.run( PDFQuestionInput( pdf=pdf, question="What are the main points from the document?" ) ) print(agent_run) # Output: # ================================================== # ================================================== # { # "answer": "This is a SEC Form 4 filing, a Statement of Changes in Beneficial Ownership, for Teri List regarding Microsoft Corporation (MSFT). The document details non-derivative securities acquired and beneficially owned, as well as derivative securities. Teri List has granted power of attorney to Julia Stark, Benjamin O. Orndorff, Michael Pressman, Keith R. Dolliver and Christyne Mayberry.", # "quotes": [ # "STATEMENT OF CHANGES IN BENEFICIAL OWNERSHIP", # "MICROSOFT CORP [ MSFT]", # "Common Stock", # "Julia Stark, Attorney-in-fact for Teri List", # "I revoke my prior Microsoft Corporation Power of Attorney.", # "The individuals who are authorized to act as my Attorney-In-Fact under this Power of Attorney are as follows:\nJulia Stark\nBenjamin O. Orndorff\nMichael Pressman\nKeith R. Dolliver\nChristyne Mayberry" # ] # } # ================================================== # Cost: $ 0.00033 # Latency: 3.61s ## From a local file pdf_path = "path/to/pdf.pdf" with open(pdf_path, "rb") as pdf_file: content = pdf_file.read() pdf = PDF(content_type="application/pdf", data=content) agent_run = await answer_pdf_question.run( PDFQuestionInput( pdf=pdf, question="What are the main points from the document?" ) ) print(agent_run) ``` {% hint style="info" %} Try this `pdf-answer-bot` with [your own PDF in the WorkflowAI playground](https://workflowai.com/docs/agents/pdf-answer-bot/1). {% endhint %} ## Audio Some LLMs can also process audio files directly, without the need to transcribe them first. First, let's use the `Audio` class to add audio as input to an agent. ```python from workflowai.fields import Audio class AudioInput(BaseModel): audio: Audio = Field() ``` Then, let's create an agent that can detect spam in audio. ```python class AudioOutput(BaseModel): is_spam: bool = Field(description="Whether the audio contains spam") explanation: str = Field(description="Explanation of the result") @workflowai.agent(id="audio-spam-detector") async def detect_audio_spam(input: AudioInput) -> AudioOutput: """ Analyze the provided audio file and determine if it contains spam. """ ... ``` {% hint style="warning" %} This part of the documentation is not yet complete. {% endhint %} Once the agent is created, you can use `agent.list_models()` to see which models support audio input. Check the `is_not_supported_reason` field to determine if a model supports your multimodal use case: ```python models = await detect_audio_spam.list_models() # Filter for supported models supported_models = [model.id for model in models if model.is_not_supported_reason is None] # Now supported_models contains all models that can process audio ``` Then, you can run the agent with an audio file. ```python # Run the agent audio_path = "path/to/audio.mp3" with open(audio_path, "rb") as audio_file: content = audio_file.read() audio = Audio(content_type="audio/mp3", data=content) agent_run = await detect_audio_spam.run( AudioInput(audio=audio), model=Model.GEMINI_2_0_FLASH_LATEST ) print(agent_run) ``` ## Video {% hint style="warning" %} Video file as input is not supported yet. {% endhint %} # Tools {% hint style="info" %} First, read our introduction to [tools](/concepts/tools). {% endhint %} Tools allow enhancing an agent's capabilities by allowing it to call external functions. Tools enable the creation of agents that can perform actions, retrieve information, and make decisions based on real-time data. ## Defining custom tools Custom tools are defined as regular python functions, and can be async or sync. ```python # Sync tool def get_current_time(timezone: Annotated[str, "The timezone to get the current time in. e-g Europe/Paris"]) -> str: """Return the current time in the given timezone in iso format""" return datetime.now(ZoneInfo(timezone)).isoformat() # Tools can also be async async def get_latest_pip_version(package_name: Annotated[str, "The name of the pip package to check"]) -> str: """Fetch the latest version of a pip package from PyPI""" url = f"https://pypi.org/pypi/{package_name}/json" async with httpx.AsyncClient() as client: response = await client.get(url) response.raise_for_status() data = response.json() return data['info']['version'] ``` To use the tool, add the function to the `tools` list in the `@workflowai.agent` decorator. {% hint style="warning" %} It must be possible to determine the schema of a tool from the function signature. This means that\ the function must have type annotations and use standard types or `BaseModel` only for now. {% endhint %} ```python @workflowai.agent( id="research-helper", tools=[get_current_time, get_latest_pip_version], model=Model.GPT_4O_LATEST, ) async def answer_question(_: AnswerQuestionInput) -> AnswerQuestionOutput: ... ``` If an agent has access to `tools`, and the model\ deems that tools are needed for a particular run, the agent will: * call all tools in parallel * wait for all tools to complete * reply to the run with the tool outputs * continue with the next step of the run, and re-execute tools if needed * ... * until either no tool calls are requested, the max iteration (10 by default) or the agent has run to completion {% hint style="info" %} The default maximum number of tool call iterations (turns) is 10. You can override this limit by passing the `max_turns` argument when calling the agent's `run` method: ```python # Allow up to 50 tool call iterations for this specific run run = await answer_question.run( AnswerQuestionInput(question="What is the current time in Phoenix, AZ?"), max_turns=50 ) ``` {% endhint %} ```python run = await answer_question( AnswerQuestionInput(question="What is the current time in Phoenix, AZ?") ) print(run) # Output: # ================================================== # { # "answer": "The current time in Phoenix, Arizona is 2:42 PM MST (Mountain Standard Time) on February 14, 2025.", # "sources": [] # } # ================================================== # Cost: $ 0.006414 # Latency: 2.64s ``` It's important to understand that there are actually two runs created in a single agent `run` call: * the first run returns an empty output with a tool call request with a timezone ``` # First run ``` * the second run returns the current time in the given timezone ``` # Second run ``` Only the last run is returned to the caller. Another example: ```python run = await answer_question(AnswerQuestionInput(question="What is the latest version of workflowai package?")) print(run) # Output: # ================================================== # { # "answer": "The latest version of the 'workflowai' package is 0.5.5.", # "sources": [ # "PyPI" # ] # } # ================================================== # Cost: $ 0.0027 # Latency: 1.38s ``` {% hint style="info" %} You can **not** directly use the web [Playground](/features/playground) to test custom tools, since the tools execution is done through your code. {% endhint %} ### Hosted tools {% hint style="warning" %} This section is not up to date. {% endhint %} WorkflowAI hosts a few tools: * `@browser-text` allows fetching the content of a web page (text-only) * `@google-search` allows performing a web search using Google's search API * `@perplexity-sonar-pro` allows performing a web search using Perplexity's Sonar Pro model Hosted tools tend to be faster because there is no back and forth between the client and the WorkflowAI API. Instead,\ if a tool call is needed, the WorkflowAI API will call it within a single request. A single run will be created for all tool iterations. To use a tool, simply add it's handles to the instructions (the function docstring): ```python @workflowai.agent(id="web-search-agent", model=Model.CLAUDE_3_5_HAIKU_LATEST) async def search_web(input: SearchWebInput) -> SearchWebOutput: """ You can use @google-search and @browser-text when relevant. """ ... ``` ```python run = await search_web( SearchWebInput( query="When was the last iPhone released?" ) ) print(run) # Output: ``` # Errors Agents can raise errors, for example when the underlying model fails to generate a response or when\ there are content moderation issues. All errors are wrapped in a `WorkflowAIError` that contains details about what happened.\ The most interesting fields are: * `code` is a string that identifies the type of error, see the [errors.py](https://github.com/WorkflowAI/python-sdk/blob/main/workflowai/core/domain/errors.py) file for more details * `message` is a human readable message that describes the error The `WorkflowAIError` is raised when the agent is called, so you can handle it like any other exception. ```python from datetime import date import workflowai from pydantic import BaseModel, Field from workflowai import Model, WorkflowAIError # define your input and output fields class Input(BaseModel): transcript: str call_date: date class Output(BaseModel): positive_points: list[str] = Field(description="List of positive points from the call", default_factory=list) negative_points: list[str] = Field(description="List of negative points from the call", default_factory=list) # define your agent @workflowai.agent(model=Model.GEMINI_2_0_FLASH_LATEST) async def analyze_call_feedback(input: Input) -> Output: """ Analyze customer call feedback and extract positive and negative points. """ ... try: await analyze_call_feedback( CallFeedbackInput( transcript="[00:01:15] Customer: The product is great!", call_date=date(2024, 1, 15) ) ) except WorkflowAIError as e: print(e.code) print(e.message) ``` #### Recoverable errors Sometimes, the LLM outputs an object that is partially valid, good examples are: * the model context window was exceeded during the generation * the model decided that a tool call result was a failure In this case, an agent that returns an output only will always raise an `InvalidGenerationError` which\ subclasses `WorkflowAIError`. However, an agent that returns a full run object will try to recover from the error by using the partial output. ```python run = await agent(input=Input(name="John")) # The run will have an error assert run.error is not None # The run will have a partial output assert run.output is not None ``` # Instructor ## Why use WorkflowAI with Instructor? WorkflowAI integrates seamlessly with your existing [Instructor](https://github.com/instructor-ai/instructor) code. Simply replace the OpenAI base URL with `run.workflowai.com/v1` and use your WorkflowAI API key instead of OpenAI API key. With this simple change, you instantly get: * **Access to over** [**100+ models**](https://workflowai.com/developers/python/instructor) **(and counting)** from OpenAI, Google, Anthropic, Llama, Grok, Mistral, etc. New models are usually added to WorkflowAI just a few hours after their public release. * **High Reliability with Automatic Fallback** thanks to our multi-provider infrastructure. For example, we fall back on Azure OpenAI when OpenAI is down. If using Claude, we fall back on AWS Bedrock when Anthropic is down. Our [uptime](https://status.workflowai.com/) for the last 5 months is 100%, and the overhead of our API is only 100ms! We are working on smart cross model fallback. (e.g. fallback on Claude 3.7 when GPT-4.1 is down) * **Guaranteed structured outputs** thanks to our native structured generation provider features (for models supporting structured generation) and thanks to our carefully crafted prompt and automatic retry (for models not supporting structured generation) * **Unlimited, free observability** visualize all your LLMs [runs](https://docs.workflowai.com/concepts/runs), share runs with your team, [evaluate](https://docs.workflowai.com/features/reviews) runs, add runs to [benchmarks](https://docs.workflowai.com/features/benchmarks) *(note that* [*templating with input variables*](#templating-with-input-variables) *is required to run benchmarks)*, [re-run input](https://docs.workflowai.com/features/playground) on different models, etc. * **Fix your agents in seconds without deploying code** Optionally, you can use our [deployment](#using-deployments-for-server-managed-instructions) features to enhance & deploy your agent's instruction right from our web-app. Ideal for fixing agent corner cases in production. *Note that* [*templating with input variables*](#templating-with-input-variables) *is a required to use deployments.* * **Zero token price markup** because we negotiate bulk deals with major providers, you will pay exactly the same price as if you were going directly to the provider. And you get a unified, detailed view of your LLM spending (per agent, per day, etc.). Also, no need for a separate key for each provider. You get your WorkflowAI API key and you can access all major providers. * **Cloud-based or self-hosted** thanks to our [open-source](https://github.com/WorkflowAI/WorkflowAI/blob/main/LICENSE) licensing model * **We value your privacy** and we are SOC-2 Type 1 certified. We do not train models on your data, nor do the LLM providers we use. Learn more about all WorkflowAI's features in our [docs](https://docs.workflowai.com/). ## 1-minute integration of WorkflowAI in existing Instructor code ### Instructor Setup (optional) If not done already, install the required packages: ```bash pip install instructor openai pydantic ``` ### WorkflowAI credentials config You can obtain your WorkflowAI API key with **$5 of free credits** [here](https://workflowai.com/developers/python/instructor/). Then either export your credentials: ```bash export WORKFLOWAI_API_KEY= export WORKFLOWAI_API_URL=https://run.workflowai.com/v1 ``` or add those to a .env: ```bash WORKFLOWAI_API_KEY= WORKFLOWAI_API_URL=https://run.workflowai.com/v1 ``` ## Simple User Info Extraction Example Here is how to extract user info from a message with WorkflowAI and instructor: ```python import os import instructor from openai import OpenAI from pydantic import BaseModel class UserInfo(BaseModel): name: str age: int def extract_user_info(user_message: str) -> UserInfo: client = instructor.from_openai( OpenAI( base_url=os.environ["WORKFLOWAI_API_URL"], # OpenAI now uses WorkflowAI's URL and API key api_key=os.environ["WORKFLOWAI_API_KEY"], # Get your API key with $5 free credits at workflowai.com/developers/python/instructor ), mode=instructor.Mode.OPENROUTER_STRUCTURED_OUTPUTS, ) return client.chat.completions.create( model="user-info-extraction-agent/gpt-4o-mini-latest", # Recommendation: use '/' format, see why in the next section. response_model=UserInfo, messages=[{"role": "user", "content": user_message}], ) if __name__ == "__main__": user_info = extract_user_info("John Doe is 32 years old.") print("Basic example result:", user_info) # UserInfo(name='John Doe', age=32) ``` We recommend using the `mode=instructor.Mode.OPENROUTER_STRUCTURED_OUTPUTS` because this mode leverages structured generation, that 100% guarantees the output object will enforce the requested `response_model`. In case the model used does not support structured generation, we'll use JSON-mode and instruct the model with the schema to enforce, in the system message. Even if the model does not support JSON-mode we'll parse a JSON from the model's raw completion and retry if the model generated a malformed JSON or a JSON that does not enforce the requested JSON schema. Note that if `mode=...` is omitted, the `TOOLS` mode will be used. Other supported modes include `TOOLS_STRICT`, `JSON` and `JSON_SCHEMA` and you should be able to obtain well-formed object with those modes, but once again `OPENROUTER_STRUCTURED_OUTPUTS` is recommended in order to leverage the native structured generation of providers when available. ### Why use `model=/` ? When specifying the `model` parameter of the `client.chat.completions.create` method, we recommend to use the `/` format. For example: * `"user-info-extraction-agent/gpt-4o-mini-latest"` Adding an `` will allow your different agents to be properly organized in your WorkflowAI account, as shown below: ![Agent list](https://2418444523-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FW4ng0K5LfFjYqHYuPgNh%2Fuploads%2Fgit-blob-b5a91a38573bb1927978d6f4e9b339a8ce5cbd5a%2Fagent-list.png?alt=media) ## Access over 100+ models, without any setup. The WorkflowAI chat completion endpoint allows you to run more than 100 models using the same endpoint schema as OpenAI chat completion, making switching model family completely transparent on your side. To change the model to use, simply update the `model` string, ex: ```python import os import instructor from openai import OpenAI from pydantic import BaseModel class UserInfo(BaseModel): name: str age: int def extract_user_info(user_message: str) -> UserInfo: client = instructor.from_openai( OpenAI(base_url=os.environ["WORKFLOWAI_API_URL"], api_key=os.environ["WORKFLOWAI_API_KEY"]), mode=instructor.Mode.OPENROUTER_STRUCTURED_OUTPUTS, ) return client.chat.completions.create( model="user-info-extraction-agent/claude-3-7-sonnet-latest", # Agent now runs Claude 3.7 Sonnet response_model=UserInfo, messages=[{"role": "user", "content": user_message}], ) if __name__ == "__main__": user_info = extract_user_info("John Black is 33 years old.") print("Basic example result:", user_info) # UserInfo(name='John Black', age=33) ``` In this case, the agent now runs on Claude 3.7 Sonnet. The complete list of our supported models is available [here](https://workflowai.com/developers/python/instructor). ## Observing your agent's runs in WorkflowAI WorkflowAI allows you to view all the runs that were made for your agent: ![Run list](https://2418444523-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FW4ng0K5LfFjYqHYuPgNh%2Fuploads%2Fgit-blob-37a529b73871152aa5ecb98a5b2816bb94423e58%2Flist-runs.png?alt=media) You can also inspect a specific run and review the run: ![Run details](https://2418444523-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FW4ng0K5LfFjYqHYuPgNh%2Fuploads%2Fgit-blob-de7d301b42cbd6ec9498a6a5e67072ba1bbc2c7d%2Frun-view.png?alt=media) ### Comparing models side-by-side In the WorkflowAI's 'Playground', you can run models side-by-side on the same input, in order to compare the model's output quality, latency and price, as shown below: ![Playground](https://2418444523-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FW4ng0K5LfFjYqHYuPgNh%2Fuploads%2Fgit-blob-b8c7dbcc28d019f1cb59734aea9ded72e8d70313%2Fplayground-fullscreen.png?alt=media) You can either re-run an input from production, manually define an input, import an input, or generate a synthetic input in the 'Playground' *(synthetic input generation requires* [*templating with Input Variables*](#templating-with-input-variables) *below)* ## Async Support You can run generation asynchronously, the same way as with the normal OpenAI implementation: ```python import os import asyncio import instructor from openai import AsyncOpenAI from pydantic import BaseModel class UserInfo(BaseModel): name: str age: int async def extract_user_info_async(user_message: str) -> UserInfo: client = instructor.from_openai( AsyncOpenAI(base_url=os.environ["WORKFLOWAI_API_URL"], api_key=os.environ["WORKFLOWAI_API_KEY"]), mode=instructor.Mode.OPENROUTER_STRUCTURED_OUTPUTS, ) return await client.chat.completions.create( model="user-info-extraction-agent/claude-3-7-sonnet-latest", response_model=UserInfo, messages=[{"role": "user", "content": user_message}], ) if __name__ == "__main__": user_info = asyncio.run(extract_user_info_async("John Black is 33 years old.")) print("Basic example result:", user_info) # UserInfo(name='John Black', age=33) ``` ## Templating with Input Variables Introducing input variables separates static instructions from dynamic content, making your agents easier to observe, since WorkflowAI logs these input variables separately. Using input variables also allows to use [benchmarks](https://docs.workflowai.com/features/benchmarks) and [deployments](https://docs.workflowai.com/features/deployments). We'll introduce a new use case to showcase this feature: classifying an email address as 'personal', 'work' or 'unsure'. You can see in the code snippet below that the instructions now contain {{}} characters to inject variables and input variables are passed separately in `extra_body['input']`. WorkflowAI's instructions templates support all [Jinja2](https://github.com/pallets/jinja/) features. ```python import os from typing import Literal import instructor from openai import OpenAI from pydantic import BaseModel class EmailAddressClassificationOutput(BaseModel): kind: Literal["personal", "work", "unsure"] def classify_email_address(email_address: str) -> EmailAddressClassificationOutput: client = instructor.from_openai( OpenAI( base_url=os.environ["WORKFLOWAI_API_URL"], api_key=os.environ["WORKFLOWAI_API_KEY"], ), mode=instructor.Mode.OPENROUTER_STRUCTURED_OUTPUTS, ) instructions = """You must classify the email address as: - 'personal' (gmail, yahoo, etc.), - 'work' (company email address) - or 'unsure'. The email address is: {{email_address}}""" return client.chat.completions.create( model="email-classifier-agent/gpt-4o-mini", response_model=EmailAddressClassificationOutput, messages=[{"role": "user", "content": instructions}], extra_body={"input": {"email_address": email_address}}, ) if __name__ == "__main__": result = classify_email_address("steve@apple.com") print(f"Classification: {result.kind}") # 'work' ``` ## Using Deployments for Server-Managed Instructions *Note that using templated instructions as explained in the previous* [*Templating with Input Variables*](#templating-with-input-variables) *section above is needed in order to use deployments.* WorkflowAI Deployments let you register your templated instructions, model and temperature in the WorkflowAI UI. You reference the registered deployment in your code by setting the `model` parameter to `/#/`. Deployment allows you to update an agent in production in seconds without needing to deploy code. This also means that anybody at your company, for example a product manager, can maintain an agent. In 'deployment' mode, you don't need to send `messages`, since WorkflowAI uses the stored instructions, and you simply pass the input variables in `extra_body['input']`. ```python import os from typing import Literal import instructor from openai import OpenAI from pydantic import BaseModel class EmailAddressClassificationOutput(BaseModel): kind: Literal["personal", "work", "unsure"] def classify_email_address_deployment(email_address: str) -> EmailAddressClassificationOutput: client = instructor.from_openai( OpenAI( base_url=os.environ["WORKFLOWAI_API_URL"], api_key=os.environ["WORKFLOWAI_API_KEY"], ), mode=instructor.Mode.OPENROUTER_STRUCTURED_OUTPUTS, ) return client.chat.completions.create( model="email-classifier-agent/#1/production", response_model=EmailAddressClassificationOutput, messages=[], # No messages needed; instructions come from the deployment extra_body={"input": {"email_address": email_address}}, ) if __name__ == "__main__": result = classify_email_address_deployment("john.doe@gmail.com") print(f"Deployment classification: {result.kind}") # 'personal' ``` ## Streaming We are currently implementing streaming on our OpenAI compatible chat completion endpoint. We'll update this documentation shortly. ## Talk with us 💌 For any question or feedback, please contact or join us on [Discord](https://workflowai.com/discord). Thank you and happy agent building!