YOLOv7-Dynamic-Batch-ONNXRUNTIME.ipynb 5.7 MB

import sys
import torch
print(f"Python version: {sys.version}, {sys.version_info} ")
print(f"Pytorch version: {torch.__version__} ")
Python version: 3.8.13 (default, Mar 28 2022, 11:38:47) 
[GCC 7.5.0], sys.version_info(major=3, minor=8, micro=13, releaselevel='final', serial=0) 
Pytorch version: 1.12.0+cu116 
!nvidia-smi
Thu Jul 28 16:34:36 2022       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.48.07    Driver Version: 515.48.07    CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   42C    P8    16W /  N/A |     13MiB /  8192MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1463      G   /usr/lib/xorg/Xorg                  4MiB |
|    0   N/A  N/A      2517      G   /usr/lib/xorg/Xorg                  4MiB |
+-----------------------------------------------------------------------------+
!wget https://github.com/WongKinYiu/yolov7/releases/download/v0.1/yolov7-tiny.pt
--2022-07-28 16:30:52--  https://github.com/WongKinYiu/yolov7/releases/download/v0.1/yolov7-tiny.pt
Resolving github.com (github.com)... 20.205.243.166
Connecting to github.com (github.com)|20.205.243.166|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/511187726/ba7d01ee-125a-4134-8864-fa1abcbf94d5?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220728%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220728T083052Z&X-Amz-Expires=300&X-Amz-Signature=81396357980573956af703230d72bdb09561f4dc2b2d8f18e599c3a89649f1de&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=511187726&response-content-disposition=attachment%3B%20filename%3Dyolov7-tiny.pt&response-content-type=application%2Foctet-stream [following]
--2022-07-28 16:30:53--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/511187726/ba7d01ee-125a-4134-8864-fa1abcbf94d5?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220728%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220728T083052Z&X-Amz-Expires=300&X-Amz-Signature=81396357980573956af703230d72bdb09561f4dc2b2d8f18e599c3a89649f1de&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=511187726&response-content-disposition=attachment%3B%20filename%3Dyolov7-tiny.pt&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.111.133, 185.199.109.133, 185.199.108.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.111.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 12639769 (12M) [application/octet-stream]
Saving to: ‘yolov7-tiny.pt’

yolov7-tiny.pt      100%[===================>]  12.05M   226KB/s    in 63s     

2022-07-28 16:31:57 (195 KB/s) - ‘yolov7-tiny.pt’ saved [12639769/12639769]

# export ONNX model for onnxruntime
!python export.py --weights ./yolov7-tiny.pt --grid --end2end --simplify \
    --topk-all 100 --iou-thres 0.65 --conf-thres 0.35 \
    --img-size 640 640 \
    --dynamic-batch \
    --max-wh 7680
!ls
Namespace(batch_size=1, conf_thres=0.35, device='cpu', dynamic=False, dynamic_batch=True, end2end=True, grid=True, img_size=[640, 640], include_nms=False, iou_thres=0.65, max_wh=7680, simplify=True, topk_all=100, weights='./yolov7-tiny.pt')
YOLOR 🚀 v0.1-74-gd77092b torch 1.12.0+cu116 CPU

Fusing layers... 
Model Summary: 200 layers, 6219709 parameters, 6219709 gradients
/home/ubuntu/miniconda3/envs/torch/lib/python3.8/site-packages/torch/functional.py:478: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at  ../aten/src/ATen/native/TensorShape.cpp:2894.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]

Starting TorchScript export with torch 1.12.0+cu116...
/home/ubuntu/work/yolo/yolov7/models/yolo.py:51: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if self.grid[i].shape[2:4] != x[i].shape[2:4]:
TorchScript export success, saved as ./yolov7-tiny.torchscript.pt

Starting ONNX export with onnx 1.12.0...
onnxruntime
/home/ubuntu/miniconda3/envs/torch/lib/python3.8/site-packages/torch/_tensor.py:1083: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations. (Triggered internally at  aten/src/ATen/core/TensorBody.h:477.)
  return self._grad
/home/ubuntu/miniconda3/envs/torch/lib/python3.8/site-packages/torch/onnx/symbolic_opset9.py:4182: UserWarning: Exporting aten::index operator of advanced indexing in opset 12 is achieved by combination of multiple ONNX operators, including Reshape, Transpose, Concat, and Gather. If indices include negative values, the exported graph will produce incorrect results.
  warnings.warn(

Starting to simplify ONNX...
ONNX export success, saved as ./yolov7-tiny.onnx
CoreML export failure: No module named 'coremltools'

Export complete (2.81s). Visualize with https://github.com/lutzroeder/netron.
builder.py	    README.md	      w2onnx.py
cfg		    requirements.txt  weights
coco		    runs	      YOLO-TensorRT8
data		    scripts	      YOLOv7-Dynamic-Batch.ipynb
detect.py	    test.py	      yolov7-tiny-batch16.onnx
export.py	    tools	      yolov7-tiny-batch16.plan
figure		    traced_model.pt   yolov7-tiny-batch1.onnx
hubconf.py	    train_aux.py      yolov7-tiny-batch1.plan
inference	    train.py	      yolov7-tiny-batch32.onnx
LICENSE.md	    usage.md	      yolov7-tiny-batch32.plan
models		    utils	      yolov7-tiny.onnx
onnx_nms_ort.ipynb  Valid_RGB	      yolov7-tiny.pt
paper		    Valid_RGB.zip     yolov7-tiny.torchscript.pt
import cv2
import time
import random
import numpy as np
import onnxruntime as ort
from PIL import Image
from pathlib import Path
from collections import OrderedDict,namedtuple
cuda = True
w = "yolov7-tiny.onnx"
imgList = [cv2.imread('inference/images/horses.jpg'),
           cv2.imread('inference/images/bus.jpg'),
           cv2.imread('inference/images/zidane.jpg'),
           cv2.imread('inference/images/image1.jpg'),
           cv2.imread('inference/images/image2.jpg'),
           cv2.imread('inference/images/image3.jpg')]
imgList*=6
imgList = imgList[:32]
providers = ['CUDAExecutionProvider', 'CPUExecutionProvider'] if cuda else ['CPUExecutionProvider']
session = ort.InferenceSession(w, providers=providers)
names = ['person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'traffic light', 
         'fire hydrant', 'stop sign', 'parking meter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 
         'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 
         'skis', 'snowboard', 'sports ball', 'kite', 'baseball bat', 'baseball glove', 'skateboard', 'surfboard', 
         'tennis racket', 'bottle', 'wine glass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 
         'sandwich', 'orange', 'broccoli', 'carrot', 'hot dog', 'pizza', 'donut', 'cake', 'chair', 'couch', 
         'potted plant', 'bed', 'dining table', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cell phone', 
         'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddy bear', 
         'hair drier', 'toothbrush']
colors = {name:[random.randint(0, 255) for _ in range(3)] for i,name in enumerate(names)}
def letterbox(im, new_shape=(640, 640), color=(114, 114, 114), auto=True, scaleup=True, stride=32):
    # Resize and pad image while meeting stride-multiple constraints
    shape = im.shape[:2]  # current shape [height, width]
    if isinstance(new_shape, int):
        new_shape = (new_shape, new_shape)

    # Scale ratio (new / old)
    r = min(new_shape[0] / shape[0], new_shape[1] / shape[1])
    if not scaleup:  # only scale down, do not scale up (for better val mAP)
        r = min(r, 1.0)

    # Compute padding
    new_unpad = int(round(shape[1] * r)), int(round(shape[0] * r))
    dw, dh = new_shape[1] - new_unpad[0], new_shape[0] - new_unpad[1]  # wh padding

    if auto:  # minimum rectangle
        dw, dh = np.mod(dw, stride), np.mod(dh, stride)  # wh padding

    dw /= 2  # divide padding into 2 sides
    dh /= 2

    if shape[::-1] != new_unpad:  # resize
        im = cv2.resize(im, new_unpad, interpolation=cv2.INTER_LINEAR)
    top, bottom = int(round(dh - 0.1)), int(round(dh + 0.1))
    left, right = int(round(dw - 0.1)), int(round(dw + 0.1))
    im = cv2.copyMakeBorder(im, top, bottom, left, right, cv2.BORDER_CONSTANT, value=color)  # add border
    return im, r, (dw, dh)
origin_RGB = []
resize_data = []
for img in imgList:
  img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
  origin_RGB.append(img)
  image = img.copy()
  image, ratio, dwdh = letterbox(image, auto=False)
  image = image.transpose((2, 0, 1))
  image = np.expand_dims(image, 0)
  image = np.ascontiguousarray(image)
  im = image.astype(np.float32)
  resize_data.append((im,ratio,dwdh))
np_batch = np.concatenate([data[0] for data in resize_data])
np_batch.shape
(32, 3, 640, 640)
outname = [i.name for i in session.get_outputs()]
outname
['output']
inname = [i.name for i in session.get_inputs()]
inname
['images']
# batch 1 infer
im = np.ascontiguousarray(np_batch[0:1,...]/255)
out = session.run(outname,{'images':im})
out
[array([[ 0.0000000e+00,  3.6190948e+02,  2.8389893e+02,  4.9353003e+02,
          3.9562683e+02,  1.7000000e+01,  9.2383915e-01],
        [ 0.0000000e+00, -1.0330048e+00,  2.6461920e+02,  2.6221228e+02,
          4.4826050e+02,  1.7000000e+01,  9.2104465e-01],
        [ 0.0000000e+00,  2.1545255e+02,  2.7048724e+02,  3.5087741e+02,
          4.1111517e+02,  1.7000000e+01,  7.6392365e-01],
        [ 0.0000000e+00, -9.6838379e-01,  2.6136188e+02,  1.2927969e+02,
          3.3445026e+02,  1.7000000e+01,  6.9170636e-01],
        [ 0.0000000e+00,  3.0596286e+02,  2.8082364e+02,  3.7849179e+02,
          3.7234799e+02,  1.7000000e+01,  4.6678147e-01]], dtype=float32)]
# batch 4 infer
im = np.ascontiguousarray(np_batch[0:4,...]/255)
out = session.run(outname,{'images':im})
out
[array([[ 0.0000000e+00,  3.6191144e+02,  2.8390021e+02,  4.9352777e+02,
          3.9562689e+02,  1.7000000e+01,  9.2384642e-01],
        [ 0.0000000e+00, -1.0345917e+00,  2.6461029e+02,  2.6221442e+02,
          4.4825989e+02,  1.7000000e+01,  9.2106003e-01],
        [ 0.0000000e+00,  2.1546349e+02,  2.7049829e+02,  3.5087964e+02,
          4.1111389e+02,  1.7000000e+01,  7.6392013e-01],
        [ 0.0000000e+00, -9.6724701e-01,  2.6136130e+02,  1.2928046e+02,
          3.3446219e+02,  1.7000000e+01,  6.9178230e-01],
        [ 0.0000000e+00,  3.0596533e+02,  2.8082968e+02,  3.7849182e+02,
          3.7233865e+02,  1.7000000e+01,  4.6649921e-01],
        [ 1.0000000e+00,  2.1281133e+02,  2.4224355e+02,  2.8546884e+02,
          5.1077960e+02,  0.0000000e+00,  8.9588410e-01],
        [ 1.0000000e+00,  1.1204082e+02,  2.3507364e+02,  2.2090996e+02,
          5.3113416e+02,  0.0000000e+00,  8.5693455e-01],
        [ 1.0000000e+00,  4.7646179e+02,  2.3513080e+02,  5.6024109e+02,
          5.1906622e+02,  0.0000000e+00,  8.5674554e-01],
        [ 1.0000000e+00,  9.0671051e+01,  1.3539163e+02,  5.3778564e+02,
          4.4476132e+02,  5.0000000e+00,  8.3615136e-01],
        [ 2.0000000e+00,  3.7307474e+02,  1.6083412e+02,  5.7743860e+02,
          4.9584790e+02,  0.0000000e+00,  7.9307902e-01],
        [ 2.0000000e+00,  2.1895935e+02,  3.5587909e+02,  2.6999200e+02,
          4.9702692e+02,  2.7000000e+01,  5.8485562e-01],
        [ 2.0000000e+00,  6.5187561e+01,  2.4052054e+02,  4.7870172e+02,
          4.9418930e+02,  0.0000000e+00,  5.7696176e-01],
        [ 3.0000000e+00,  5.3027496e+00,  6.2176086e+01,  3.7205682e+02,
          5.8482434e+02,  0.0000000e+00,  8.7524450e-01],
        [ 3.0000000e+00,  2.6052777e+02,  1.4254730e+02,  6.3039929e+02,
          5.8130194e+02,  0.0000000e+00,  8.1194180e-01]], dtype=float32)]
# batch 6 infer
im = np.ascontiguousarray(np_batch[0:6,...]/255)
out = session.run(outname,{'images':im})
out
[array([[ 0.0000000e+00,  3.6191074e+02,  2.8389581e+02,  4.9353012e+02,
          3.9562750e+02,  1.7000000e+01,  9.2385465e-01],
        [ 0.0000000e+00, -1.0335236e+00,  2.6461029e+02,  2.6221289e+02,
          4.4826691e+02,  1.7000000e+01,  9.2106324e-01],
        [ 0.0000000e+00,  2.1546918e+02,  2.7049741e+02,  3.5089246e+02,
          4.1111246e+02,  1.7000000e+01,  7.6381278e-01],
        [ 0.0000000e+00, -9.6646881e-01,  2.6135922e+02,  1.2928015e+02,
          3.3445938e+02,  1.7000000e+01,  6.9177818e-01],
        [ 0.0000000e+00,  3.0595779e+02,  2.8082639e+02,  3.7848483e+02,
          3.7233902e+02,  1.7000000e+01,  4.6637228e-01],
        [ 1.0000000e+00,  2.1281424e+02,  2.4223289e+02,  2.8546671e+02,
          5.1078430e+02,  0.0000000e+00,  8.9587235e-01],
        [ 1.0000000e+00,  4.7646426e+02,  2.3512268e+02,  5.6024048e+02,
          5.1906110e+02,  0.0000000e+00,  8.5685480e-01],
        [ 1.0000000e+00,  1.1203514e+02,  2.3505554e+02,  2.2091090e+02,
          5.3114307e+02,  0.0000000e+00,  8.5680377e-01],
        [ 1.0000000e+00,  9.0665085e+01,  1.3541930e+02,  5.3778650e+02,
          4.4475671e+02,  5.0000000e+00,  8.3618683e-01],
        [ 2.0000000e+00,  3.7307629e+02,  1.6083318e+02,  5.7743591e+02,
          4.9584601e+02,  0.0000000e+00,  7.9310930e-01],
        [ 2.0000000e+00,  2.1895792e+02,  3.5587857e+02,  2.6999387e+02,
          4.9703476e+02,  2.7000000e+01,  5.8482271e-01],
        [ 2.0000000e+00,  6.5209000e+01,  2.4051682e+02,  4.7865540e+02,
          4.9418790e+02,  0.0000000e+00,  5.7698834e-01],
        [ 3.0000000e+00,  5.2892609e+00,  6.2162445e+01,  3.7209552e+02,
          5.8483594e+02,  0.0000000e+00,  8.7545133e-01],
        [ 3.0000000e+00,  2.6051691e+02,  1.4254585e+02,  6.3040662e+02,
          5.8132233e+02,  0.0000000e+00,  8.1180179e-01],
        [ 4.0000000e+00,  4.2224957e+02,  1.0267369e+02,  6.3170001e+02,
          5.5488086e+02,  0.0000000e+00,  9.4197059e-01],
        [ 4.0000000e+00,  4.9545387e+01,  2.7456857e+02,  1.8946579e+02,
          4.1091318e+02,  3.2000000e+01,  9.3334389e-01],
        [ 4.0000000e+00,  2.6599213e+01,  1.0745810e+02,  4.5867126e+02,
          5.5153650e+02,  0.0000000e+00,  8.2260609e-01],
        [ 5.0000000e+00,  1.9983047e+02,  3.6107239e+01,  5.2495667e+02,
          4.8967474e+02,  1.7000000e+01,  8.4681529e-01],
        [ 5.0000000e+00,  1.1123685e+02,  3.1816605e+02,  3.7944864e+02,
          5.6111890e+02,  1.6000000e+01,  6.9891703e-01]], dtype=float32)]
# batch 32 infer
im = np.ascontiguousarray(np_batch/255)
out = session.run(outname,{'images':im})[0]
for i,(batch_id,x0,y0,x1,y1,cls_id,score) in enumerate(out):
    if batch_id >= 6:
        break
    image = origin_RGB[int(batch_id)]
    ratio,dwdh = resize_data[int(batch_id)][1:]
    box = np.array([x0,y0,x1,y1])
    box -= np.array(dwdh*2)
    box /= ratio
    box = box.round().astype(np.int32).tolist()
    cls_id = int(cls_id)
    score = round(float(score),3)
    name = names[cls_id]
    color = colors[name]
    name += ' '+str(score)
    cv2.rectangle(image,box[:2],box[2:],color,2)
    cv2.putText(image,name,(box[0], box[1] - 2),cv2.FONT_HERSHEY_SIMPLEX,0.75,[225, 255, 255],thickness=2)
Image.fromarray(origin_RGB[0])
Image.fromarray(origin_RGB[1])
Image.fromarray(origin_RGB[2])
Image.fromarray(origin_RGB[3])
Image.fromarray(origin_RGB[4])
Image.fromarray(origin_RGB[5])