{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
" \n",
" \n",
" ![](\"https://user-images.githubusercontent.com/25985824/104791629-6e618700-5769-11eb-857f-d176b37d2496.png\") \n",
" Try in Google Colab\n",
" \n",
" | \n",
" \n",
" \n",
" ![](\"https://user-images.githubusercontent.com/25985824/104791634-6efa1d80-5769-11eb-8a4c-71d6cb53ccf0.png\") \n",
" Share via nbviewer\n",
" \n",
" | \n",
" \n",
" \n",
" ![](\"https://user-images.githubusercontent.com/25985824/104791633-6efa1d80-5769-11eb-8ee3-4b2123fe4b66.png\") \n",
" View on GitHub\n",
" \n",
" | \n",
" \n",
" \n",
" ![](\"https://user-images.githubusercontent.com/25985824/104792428-60f9cc00-576c-11eb-95a4-5709d803023a.png\") \n",
" Download notebook\n",
" \n",
" | \n",
"
\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# **FiftyOne Heatmaps**"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Learning how to use heatmaps in FiftyOne and a pose estimation example"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### A basic heatmap"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
",\n",
" 'heatmap2': ,\n",
"}>\n"
]
}
],
"source": [
"import cv2\n",
"import numpy as np\n",
"\n",
"import fiftyone as fo\n",
"\n",
"# Example heatmap\n",
"map_path = \"/tmp/heatmap.png\"\n",
"map = np.random.randint(256, size=(128, 128), dtype=np.uint8)\n",
"cv2.imwrite(map_path, map)\n",
"\n",
"sample = fo.Sample(filepath=\"/path/to/image.png\")\n",
"sample[\"heatmap1\"] = fo.Heatmap(map_path=map_path)\n",
"sample[\"heatmap2\"] = fo.Heatmap(map=map)\n",
"\n",
"print(sample)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Setting up [SWAHR-HumanPose](https://github.com/greatlog/SWAHR-HumanPose/tree/master) Model"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It is highly suggested that you use an venv such as conda to set up a seperate enviroment for the following. Refer to [this README](https://github.com/HRNet/HigherHRNet-Human-Pose-Estimation/blob/master/README.md) for trouble shooting"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!git clone https://github.com/greatlog/SWAHR-HumanPose.git"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Download the model linked [here](https://drive.google.com/drive/folders/13FFvwK7bDZLD4H_toueopbLhJqFjimlu), I used pose_higher_hrnet_w32_512.pth, make sure to place in the ./models dir in the SWAHR repo"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# COCOAPI=/path/to/clone/cocoapi\n",
"!git clone https://github.com/cocodataset/cocoapi.git $COCOAPI\n",
"!cd $COCOAPI/PythonAPI\n",
"# Install into global site-packages\n",
"!make install\n",
"# Alternatively, if you do not have permissions or prefer\n",
"# not to install the COCO API into global site-packages\n",
"!python3 setup.py install --user"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# CROWDPOSE=/path/to/clone/crowdpose\n",
"!git clone https://github.com/Jeff-sjtu/CrowdPose.git $CROWDPOSE\n",
"!cd $CROWDPOSE/PythonAPI\n",
"# Install into global site-packages\n",
"!make install\n",
"# Alternatively, if you do not have permissions or prefer\n",
"# not to install the CROWDPOSE API into global site-packages\n",
"!python3 setup.py install --user\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Finally you may need to roll back some packages, namely numpy"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!pip3 install numpy==1.20.0"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Generating Heatmaps"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We will use the [tools/dist_inference.py](https://github.com/greatlog/SWAHR-HumanPose/blob/master/tools/dist_inference.py) script to generate our heatmaps"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### File changes"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Make these changes to the appropriate files:"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"#Line 42 in tools/dist_inference.py add make_heatmaps\n",
"from utils.vis import draw_skeleton, make_heatmaps"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#Line 196 in tools/dist_inference.py add the following block\n",
" images,heatmaps = make_heatmaps(image,final_heatmaps[0])\n",
" master_heatmap = heatmaps[0]\n",
" for x in heatmaps:\n",
" master_heatmap = np.maximum(master_heatmap, x)\n",
" resized_m_heatmap = cv2.resize(master_heatmap,(image.shape[1],image.shape[0]))\n",
" cv2.imwrite(\n",
" os.path.join(save_dir, \"{}_heatmap.png\".format(image_name)),\n",
" resized_m_heatmap\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#Line 84 in lib/utils/vis.py add a extra return variable heatmaps\n",
"return image_grid, heatmaps"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"With our model fully set up to generate and save some heatmaps, we can navigate to the SWAHR repo and run the following command"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#In terminal in SWAHR repo\n",
"!python3 tools/dist_inference.py --world_size 1 --img_dir ~/fiftyone/quickstart/data/\n",
"--save_dir output_quick --cfg experiments/coco/higher_hrnet/w32_512_adam_lr1e-3.yaml\n",
"TEST.MODEL_FILE models/pose_higher_hrnet_w32_512.pth TEST.SCALE_FACTOR '[0.5, 1.0, 2.0]'"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Lets load our heatmaps in and visualize!"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Dataset already downloaded\n",
"Loading 'quickstart'\n",
" 100% |█████████████████| 200/200 [2.1s elapsed, 0s remaining, 96.2 samples/s] \n",
"Dataset 'quickstart' created\n"
]
},
{
"data": {
"text/html": [
"\n",
"\n",
"\n",
"\n",
"
\n",
" \n",
"
\n",
"
\n",
"