-
Notifications
You must be signed in to change notification settings - Fork 20
Open
Description
Thanks for creating this library. It greatly simplifies calling the Gradient API. I am wondering how we can add something to the streaming chunk. Our current code, just plain javascript, is below and we are trying to simplify it using this SDK, but we are not sure how to add the "type": "sources" to the stream, and this is essential for our front end RAG application. Can you please let me know how we can use the Stream part of the SDK to do something like below? Specifically, it's this line we are not sure how to implement with the SDK: controller.enqueue(textEncoder.encode(JSON.stringify({ type: "sources", data: usedSources })));
Current code
const aiRequest = await fetch(
`${AGENT_ENDPOINT}/api/v1/chat/completions`,
{
headers: {
'Content-Type': 'application/json',
'Authorization': `Bearer ${AGENT_KEY}`
},
method: 'POST',
body: JSON.stringify({
messages: [
{
role: "user",
content: aiContent //the prompt
}
],
stream: true,
include_functions_info: false,
include_retrieval_info: false,
include_guardrails_info: false
}),
}
)
const stream = new ReadableStream({
async start(controller) {
const reader = aiRequest?.body?.getReader();
if (!reader) {
controller.error("ReadableStream reader not available");
return;
}
const decoder = new TextDecoder();
while (true) {
const { done, value } = await reader.read();
if (done) { break; }
const chunk = decoder.decode(value, { stream: true });
// Split SSE lines
for (const line of chunk.split("\n")) {
if (!line.startsWith("data: ")) { continue; }
const data = line.replace("data: ", "").trim();
if (data === "[DONE]") { continue; }
try {
const json = JSON.parse(data);
const token = json.choices?.[0]?.delta?.content;
if (token) {
controller.enqueue(new TextEncoder().encode(token));
}
} catch {
continue;
}
}
}
const usedSources = items.slice(0, 5); // example: top 5 sources used
controller.enqueue(textEncoder.encode(JSON.stringify({ type: "sources", data: usedSources })));
controller.close();
},
});
//console.log ("stream", stream)
return new Response(stream, {
headers: { "Content-Type": "text/plain; charset=utf-8" },
});
Metadata
Metadata
Assignees
Labels
No labels