Skip to content

how to add to the streaming response? #21

@osseonews

Description

@osseonews

Thanks for creating this library. It greatly simplifies calling the Gradient API. I am wondering how we can add something to the streaming chunk. Our current code, just plain javascript, is below and we are trying to simplify it using this SDK, but we are not sure how to add the "type": "sources" to the stream, and this is essential for our front end RAG application. Can you please let me know how we can use the Stream part of the SDK to do something like below? Specifically, it's this line we are not sure how to implement with the SDK: controller.enqueue(textEncoder.encode(JSON.stringify({ type: "sources", data: usedSources })));

Current code

const aiRequest = await fetch(
                `${AGENT_ENDPOINT}/api/v1/chat/completions`,
                {
                    headers: {
                        'Content-Type': 'application/json',
                        'Authorization': `Bearer ${AGENT_KEY}`
                    },
                    method: 'POST',
                    body: JSON.stringify({
                        messages: [
                            {
                                role: "user",
                                content: aiContent //the prompt
                            }
                        ],
                
                        stream: true,
                        include_functions_info: false,
                        include_retrieval_info: false,
                        include_guardrails_info: false
                    }),
                }
            )

const stream = new ReadableStream({
            async start(controller) {
                const reader = aiRequest?.body?.getReader();

                if (!reader) {
                    controller.error("ReadableStream reader not available");
                    return;
                }
                const decoder = new TextDecoder();

                while (true) {
                    const { done, value } = await reader.read();
                    if (done) { break; }

                    const chunk = decoder.decode(value, { stream: true });

                    // Split SSE lines
                    for (const line of chunk.split("\n")) {
                        if (!line.startsWith("data: ")) { continue; }

                        const data = line.replace("data: ", "").trim();
                        if (data === "[DONE]") { continue; }

                        try {
                            const json = JSON.parse(data);
                            const token = json.choices?.[0]?.delta?.content;
                            if (token) {
                                controller.enqueue(new TextEncoder().encode(token));
                            }
                        } catch {
                            continue;
                        }
                    }
                }
                const usedSources = items.slice(0, 5); // example: top 5 sources used
                controller.enqueue(textEncoder.encode(JSON.stringify({ type: "sources", data: usedSources })));
                controller.close();
            },
        });
        //console.log ("stream", stream)
        return new Response(stream, {
            headers: { "Content-Type": "text/plain; charset=utf-8" },
        });

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions