Popular Posts

Popular Content

Powered by Blogger.

Search This Blog

Follow on Google+

Recent Posts

About us

August 23, 2023

Hi HN,

Gentrace is our new evaluation and observability tool for generative AI (open beta).

Generative pipelines are hard to evaluate because outputs are subjective. Lots of developers end up just doing “gut checks” on a few inputs before shipping changes, or they build up a spreadsheet of test cases that they manually run through the pipeline. Some companies outsource filling out the spreadsheet. However, in any of these cases, you end up with a very slow and expensive process for evaluation.

At one point, we did this too. Gentrace is the result of a pivot; it was an internal tool we used to automatically grade new PRs as developers shipped changes to generative pipelines that other people thought might be useful.

Gentrace makes pre-production testing of generative pipelines continuous and nearly instantaneous. In Gentrace, you:

- Import and/or construct suites of test data - Use a combination of AI and heuristic evaluators to grade for quality, hallucination, safety, etc - Use our interface to correct automated grades or add your own (yourself or a member of your team)

Gentrace integrates at a code level for evaluation, meaning we test your generative AI pipeline the way you would test normal code. This allows you to test more than just prompt changes; for example, you can compare models (eg Claude 2 vs GPT-4 vs GPT 3.5 vs Llama 2) or see the effects of additional chained steps (”Rewrite the previous answer in the following tone:”).

Here’s a video overview that goes into a bit more detail: https://youtu.be/XxgDPSrTWIw

In production, Gentrace observes for speed, cost, and data flow. It also shows real user feedback as well. We do this by integrating via our SDK at a code level; Gentrace does not proxy requests.

Soon, we’ll allow you to convert production data into test cases, allowing customer support to turn bad production generations into “failing tests” for AI teams to make pass.

We process interim steps and multiple outputs as well, helping evaluate agent flows / chains where the “last output” isn’t always the only thing that matters.

There’s been a lot of observability tools published recently. We differ from those by focusing more strongly on blending observability with strong evaluation and by using an SDK rather than a “man-in-the-middle” approach to capturing data (ie Gentrace can be down and your request to OpenAI will still succeed).

Within the evaluation landscape, we differentiate by integrating with code (see above for benefits) for capturing generative outputs and by providing a customizable UI workflow for building evaluators. In Gentrace, you start with off-the-shelf automated evaluators and then customize them to your specific task. You also build and run new evaluators on old generative outputs. Finally, you easily override automated evaluators and/or blend automated evaluation with evaluation by humans on your team.

We also focus on being suitable for business use. We are SOC 2 Type 1 compliant (Type 2 coming shortly), have robust legal documentation around data processing, security, and privacy, and have already passed several vendor legal and security reviews at large technology companies.

Our standard usage-based pricing is available on the website: https://gentrace.ai/pricing

If you are building features with generative AI, we would love to get your feedback. You can self-serve sign up (without a credit card) for a 14 day trial here: https://gentrace.ai/

We’re available right here for feedback and questions. We’re also available at support@gentrace.ai.

Best, Doug, Vivek, and Daniel

Comments URL: https://news.ycombinator.com/item?id=37238648

Points: 9

# Comments: 0



from Hacker News: Front Page https://gentrace.ai
Continue Reading
August 20, 2023

Hi everyone, nice to meet you and I am a newcomer of HN.

I have made a binary tool Aih that could communicate with Bard, ChatGPT, Claude, and Llama(HuggingChat) from the terminal. https://github.com/Databingo/aih

Since CAPTCHA challenges and bots detecting have become increasingly difficult, I've changed my strategy from hacking the APIs to simulating a real browser's action. The tool first takes the logged-in cookies of Google, ChatGPT, Claude, and HuggingChat accounts from the real Chrome browser, then it opens an invisible instance of Chromium for communication, then displays the answers in terminal.

I think it's useful especially when I am researching some topics and need to compare answers of those AI models at the same time.

Feel free to test and welcome provide feedback!

Comments URL: https://news.ycombinator.com/item?id=37198357

Points: 7

# Comments: 0



from Hacker News: Front Page https://ift.tt/rLtoY43
Continue Reading
August 02, 2023

Hi guys,

Hope everyone is well. This app was borne out of my own frustration. I thought that I was terrible at learning languages at school, since I didn't become conversational in French after 5 years of study. However, I later traveled with some French friends and, in just under 3 weeks, I was able to hold a reasonable conversation. I realized that there's no substitute for speaking to native speakers.

I tried to adopt this approach for other languages, but it's much harder to find people to practise with when you aren't travelling. I started using iTalki to meet people from different countries and chat to them. It quickly became very expensive and time-consuming to schedule the calls, so I gave up.

I made PrettyPolly so that anyone can easily practice speaking 26 languages orally. The app uses ChatGPT (amongst other tools) to allow you to practice speaking whenever you want. It also generates a fluency score for each conversation so that you have an objective way of tracking progress.

It's free to use (up to 15 conversations per month). I've found that using it once or twice per day is plenty, and you'll be amazed at how much you will pick up in a week. I've added some FAQs here in case useful - https://www.prettypolly.app/learn

Would really appreciate any feedback. Let me know if you have any questions, issues or suggestions.

Thanks, Chris

Comments URL: https://news.ycombinator.com/item?id=36973400

Points: 8

# Comments: 4



from Hacker News: Front Page https://ift.tt/2PlQkZt
Continue Reading
Subscribe to: Posts (Atom)