Run GPT-OSS-20B with Advanced Inference in Colab
Load OpenAI's 40GB GPT-OSS-20B model in Colab on T4 GPU using MXFP4 quantization and torch.bfloat16; implement reasoning controls, JSON schemas, multi-turn memory, streaming, tools, and batch processing for production workflows.
GPU and Dependency Setup for Reliable Loading
GPT-OSS-20B requires ~16GB VRAM (T4/A100 recommended) and downloads ~40GB on first run. Install transformers>=4.51.0, accelerate, sentencepiece, protobuf, huggingface_hub, gradio, ipywidgets, and openai-harmony. Verify CUDA with torch.cuda.is_available() and check memory via torch.cuda.get_device_properties(0).total_memory. Load via AutoModelForCausalLM.from_pretrained("openai/gpt-oss-20b", torch_dtype=torch.bfloat16, device_map="auto", trust_remote_code=True) and AutoTokenizer. Use pipeline("text-generation") for inference. Post-load, expect ~allocated/reserved GPU memory printouts to confirm ~16GB usage. OpenAI recommends temperature=1.0, top_p=1.0; adjust to 0.8 for consistency.
Basic generation: Pass messages list like {'role': 'user', 'content': 'query'} to pipeline with max_new_tokens=256, do_sample=True, pad_token_id=tokenizer.eos_token_id. Extracts response from output[0]["generated_text"][-1]"content". Handles QA, code gen, creative tasks effectively.
Configurable Reasoning and Structured Outputs
Define ReasoningEffortController with three levels:
- Low: "Be concise", max_tokens=200, temp=0.7 → quick answers.
- Medium: "Think step-by-step", max_tokens=400, temp=0.8 → balanced.
- High: Multi-step CoT prompt, max_tokens=800, temp=1.0 → deep analysis. Prepend system prompt to messages; scales token budget and detail for logic puzzles, improving accuracy on complex queries.
For JSON: StructuredOutputGenerator enforces schema via strict system prompt ("ONLY output valid JSON matching schema, no markdown"). Cleans response (strip ```json blocks), parses with json.loads(), retries up to 2x on JSONDecodeError by appending error feedback. Examples: Entity extraction schema {'name': 'str', 'type': 'str', 'description': 'str', 'key_facts': 'str'}; recipe schema with prep_time_minutes (int), ingredients list of dicts. Reduces hallucinations, ensures type safety for APIs.
Stateful Chats, Streaming, Tools, and Batch Efficiency
ConversationManager maintains history list, prepends system + history to each chat() call (max_new_tokens=300, temp=0.8). Supports get_history_length(), clear_history(), context_summary(). Enables memory across turns, e.g., recalling user name/field.
Streaming: Use TextIteratorStreamer(tokenizer, skip_prompt=True) with model.generate(inputs from tokenizer.apply_chat_template(), streamer=streamer, max_new_tokens=200) in thread. Prints tokens live, reveals decoding speed/behavior.
Tools via ToolExecutor: Decorator @register(name, desc) for funcs like calculator (safe eval with math whitelist), get_time(), simulated weather/search. Prompt lists tools; model outputs "TOOL: name\nARGS: json". Parse, execute, feed result back for final response. Loops once for math/time/weather queries.
Batch: batch_generate(prompts, batch_size=2) processes in chunks via pipeline on list of message lists. Handles 5+ prompts efficiently, e.g., trivia QA, cutting per-call overhead for throughput testing.