TokScope is a tool for visualizing token embeddings from NLP models. It uses t-SNE to show relationships and clusters among words or subwords based on semantic similarity. You can explore token relationships, identify semantic clusters, and analyze distribution patterns. Features include interactive exploration, viewing detailed statistics, and customization options for data representation.
Features
Interactive Exploration
Allows users to explore token relationships in a visual format where tokens appear as points on a scatter plot using t-SNE for dimensionality reduction.
Detailed Views
Provides options to view selected token details with histogram views, token lists, and categories.
Data Statistics
Displays pie or bar charts showing the distribution of different token types as per the tokenizer.
Customization
Enables users to adjust opacity and switch color palettes for better visualization of data clusters.