leavelet/cutlass
1
0
Fork 0
Code Issues Pull Requests Actions 25 Packages Projects Releases Wiki Activity
623 Commits 1 Branch 1 Tag
5e497243f7ad13a2aa842143f9b10bbb23d98292
Commit Graph
114 Commits
Author SHA1 Message Date
Yujia Zhaiandyuzhai 79fc51f4b8 v3.9 update (#2213) 
Co-authored-by: yuzhai <yuzhai@nvidia.com>
 2025-04-03 02:10:16 -04:00
Yujia Zhaiandyuzhai 6f4921858b v3.9 update (#2203) 
* v3.9 update

* voidD

---------

Co-authored-by: yuzhai <yuzhai@nvidia.com>
 2025-04-02 15:11:18 -04:00
Yujia ZhaiyuzhaiHaicheng WuHaicheng Wu
 		62750a2b75 v3.9 (#2185) 
* v3.8 update x

* fix blackwell gg

* doc change

* doc change

* doc change

---------

Co-authored-by: yuzhai <yuzhai@nvidia.com>
Co-authored-by: Haicheng Wu <haichengw@nvidia.com>
Co-authored-by: Haicheng Wu <57973641+hwu36@users.noreply.github.com>
 2025-03-21 01:52:23 -04:00
Jack Kosaian 6c6b78550e Fix SM90 beta=1 hang and stream-K launch errors (#2172) 
* Fix stream-K occupancy calculation

* Fix beta=1 hang
 2025-03-13 14:07:37 -04:00
dePaul MilleranddePaul Miller 06e560d98a Blockwise/Groupwise kernel improvement and programatic dependent launch enablement (#2161) 
Co-authored-by: dePaul Miller <23461061+depaulmillz@users.noreply.github.com>
 2025-03-10 14:36:11 -04:00
Lucas Wilkinson df18f5e4f5 Improvements for: Groupwise scaling along M for FP8 gemm (#2095) 
* fix blockwise fp8 kernels

Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>

* wip, < 128 not working

Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>

* fix < 128

Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>

* reduce diff

Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>

* review comments

Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>

* support partial n blocks

Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>

* fix build errors

Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>

---------

Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
 2025-02-27 22:39:29 -05:00
dePaul MilleranddePaul Miller ca4fdbea70 Blockwise and Groupwise GEMM for Blackwell and Improvements for Hopper (#2139) 
- Blockwise and Groupwise GEMM improvements for Hopper.
- Blockwise and Groupwise GEMM for Blackwell.
- Blockwise Grouped GEMM for Hopper.
- Static ScalePromotionInterval for Hopper FP8 GEMMs.

Co-authored-by: dePaul Miller <23461061+depaulmillz@users.noreply.github.com>
 2025-02-26 12:44:58 -05:00
ANIKET SHIVAMandHaicheng Wu 9b3772dfa6 Hopper Grouped GEMM support for FP8 Accum (#2123) 
* Add support for fp8accum, with profiler extension

* Update .gitignore

* contri

---------

Co-authored-by: Haicheng Wu <haichengw@nvidia.com>
 2025-02-20 21:55:26 -05:00
Yujia Zhaiandyuzhai b84e9802d8 update 3.8 v2 (#2112) 
* update 3.8 v2

* update 3.8

---------

Co-authored-by: yuzhai <yuzhai@nvidia.com>
 2025-02-19 22:03:14 -05:00
Yujia Zhaiandyuzhai 833f6990e0 v3.8.0 update (#2082) 
* 3.8 update

* fix Markus' name

---------

Co-authored-by: yuzhai <yuzhai@nvidia.com>
 2025-02-06 21:33:40 -05:00
LiangzlHaicheng Wu
 		3c28697b9f Groupwise scaling along M for FP8 gemm (#2037) 
* FP8 groupwise scaling along M

* small updates

---------

Co-authored-by: zl <zl@deepseek.com>
Co-authored-by: Haicheng Wu <haichengw@nvidia.com>
 2025-01-31 13:51:28 -05:00
mihir-awatramaniHaicheng WuHaicheng Wu
 		389e493055 CUTLASS 3.8 Release (#2059) 
* CUTLASS 3.8 Release

* update

* Update README.md

* Revert "Update README.md"

This reverts commit b353e36fe83e0815f99b44e46c0c95494c44726b.

* update

* update

---------

Co-authored-by: Haicheng Wu <57973641+hwu36@users.noreply.github.com>
Co-authored-by: Haicheng Wu <haichengw@nvidia.com>
 2025-01-25 02:44:06 -05:00
Yujia ZhaiyuzhaiHaicheng Wu
 		b78588d163 CUTLASS 3.7 (#2045) 
* CUTLASS 3.7

* clean up changelog

---------

Co-authored-by: yuzhai <yuzhai@nvidia.com>
Co-authored-by: Haicheng Wu <haichengw@nvidia.com>
 2025-01-18 09:53:07 -05:00
Manish GuptaandHaicheng Wu ef5620dd1d Blockwise Scaling for FP8 (#1932) 
* F8 Blockwise Scaling

* two more NumProducerThreadEvents

---------

Co-authored-by: Haicheng Wu <haichengw@nvidia.com>
 2025-01-09 11:22:09 -05:00
Yujia ZhaiyuzhaiHaicheng Wu
 		3d261a5974 3.6.0 update (#2005) 
* 3.6.0 update

* doc and swap stuff

---------

Co-authored-by: yuzhai <yuzhai@nvidia.com>
Co-authored-by: Haicheng Wu <haichengw@nvidia.com>
 2024-12-25 01:34:40 -05:00
Lain 4c42f73fda Improve mixed dtype GEMM (#1972) 
* update

* fix a typo
 2024-12-06 13:33:22 -05:00
LainandSiyuan Fu 8aa95dbb88 Fix the racing condition of mixed-input gemm when writing the registers (#1931) 
* move two warpgroup_wait

* merge main

---------

Co-authored-by: Siyuan Fu <siyuanf@nvidia.com>
 2024-11-08 13:15:54 -05:00
chenwei 19f51596e8 feat: support kFactor 8 used in mma tensor op tile iterator (#1512) 2024-10-29 11:56:59 -04:00
azhurkevich e8a8b69365 Refactor some GroupedGEMM logic (#1899) 2024-10-25 20:14:01 -04:00
LainandSiyuan Fu be692b48b0 remove redundant hardcoded packing configs in mixed dtype gemm (#1894) 
Co-authored-by: Siyuan Fu <siyuanf@nvidia.com>
 2024-10-23 14:24:09 -04:00
Xinyu Yang b0c09ed077 fix by adding public (#1753) 2024-10-23 12:45:58 -04:00
Sergey Klevtsov 08101d9d0c Improve sm90 mixed dtype kernel (#1883) 2024-10-17 20:06:38 -04:00
Haicheng Wu 755194a7bd add is_last_tile 2024-10-17 12:11:02 -07:00
Yujia ZhaiyuzhaiHaicheng Wu
 		cc3c29a81a CUTLASS 3.6.0 (#1850) 
* v3.6

* update changelog

* update readme

* fix typo

* fixing typos

* hopper gemm with weight prefetch

---------

Co-authored-by: yuzhai <yuzhai@nvidia.com>
Co-authored-by: Haicheng Wu <haichengw@nvidia.com>
 2024-10-09 15:33:27 -04:00
Chenggang Zhao 1ebda1ccef Fix MMA promotion interval assertions (#1641) 2024-09-16 12:38:42 -04:00
Junkai-Wu dbdae514e0 Support for TMA Epilogue for Group Gemm and add pingpong ptr array & Group Gemm (#1795) 2024-09-11 00:07:31 -04:00
Aleksandar SamardžićAleksandar SamardžićHaicheng Wu
 		e1976daacc Add support for mixed 4-bit/8-bit data types GEMM (#1413) 
* Add support for mixed 4-bit/8-bit data types GEMM

* fix ( and )

---------

Co-authored-by: Aleksandar Samardžić <asamardzic@matf.bg.ac.rs>
Co-authored-by: Haicheng Wu <haichengw@nvidia.com>
 2024-08-29 23:11:06 -04:00
dePaul Miller 06b21349bc 1x1x1 cluster launch (#1673) 2024-08-01 12:20:28 -04:00
Vijay Thakkar be60a0b272 CUTLASS 3.5.1 (#1623) 
* CUTLASS 3.5.1

* updates, optimizations, fixes
 2024-07-29 08:46:24 -04:00
Chengquan Jiang 56b46e2d13 Fix grouped gemm invalid memory access to problem shapes (#1543) 2024-07-10 11:55:22 -04:00
Vijay Thakkar 7d49e6c7e2 Updates for CUTLASS 3.5.0 (#1468) 2024-04-11 21:33:40 -04:00
Vijay Thakkar 629f4653c3 CUTLASS 3.5.0 (#1411) 2024-03-19 17:51:04 -04:00
ANIKET SHIVAM bbe579a9e3 Updates for CUTLASS 3.4.1 (#1346) 
* Updates for CUTLASS 3.4.1

* minor epi change
 2024-02-15 15:48:34 -05:00
Aleksandar SamardžićandAleksandar Samardžić ca37d632c9 Remove sparse GEMM with row broadcasted bias vector (#1302) 
This reverts commit d3e72719b4.

Co-authored-by: Aleksandar Samardžić <asamardzic@matf.bg.ac.rs>
 2024-01-17 14:06:27 -05:00
ANIKET SHIVAM 751eb9a885 Update license year (#1306) 2024-01-16 14:37:22 -05:00
ANIKET SHIVAM 2f589ffa76 Updates for 3.4 release. (#1305) 2024-01-16 13:42:51 -05:00
Aleksandar Samardžić 5c756eb774 Add support for sparse GEMM with visitor epilogue (#1189) 
* Add support for sparse GEMM with visitor epilogue

* Refactor changes at the kernel level
 2024-01-04 12:38:11 -05:00
Pradeep RamaniandPradeep Ramani 8236f30675 CUTLASS 3.4.0 (#1286) 
* CUTLASS 3.4.0

* Update CHANGELOG.md

---------

Co-authored-by: Pradeep Ramani <prramani@nvidia.com>
 2023-12-29 15:21:31 -05:00
Christian SiggandHaicheng Wu e1483d5fa0 Collection of changes to fix clang build. (#1200) 
* Remove unused variables

* Qualify calls to make_fragment_? from templated base class.

Fixes clang build error.

* Add missing `#include <cstdio>`

* Various changes to fix clang compile errors.

* More changes to fix clang build.

Remaining issues:

- `params` initializer of `CollectiveEpilogue`.
- `ops` initializer of `Sm90VisitorImplBase`.
- `__usAtomicCAS` needs to be added to clang upstream.

* Fix remaining clang build issues.

* Qualify `cute::rank()` calls.

* Qualify some more calls that are otherwise ambiguous between `cute` and `std` namespace.

* Double-escape special registers in inline asm.

* small change

---------

Co-authored-by: Haicheng Wu <haichengw@nvidia.com>
 2023-12-08 14:42:12 -05:00
Ali HassaniandAli Hassani f4a0216601 Fix bug in single source GEMM with residual + streamk (#1249) 
Followup to #1224.

A change in the stream-k threadblock swizzle ctor since 3.3 breaks
single source GEMM with fused epilogue and stream-k. Multi-source was
already corrected.

Co-authored-by: Ali Hassani <ahassanijr@gmail.com>
 2023-12-07 11:12:02 -05:00
Pradeep Ramani e9e30c2304 Updates and Bug fixes to CUTLASS 3.3 (#1232) 2023-12-05 09:50:49 -05:00
cyyeverandHaicheng Wu 10b850f9c7 Fix some sign conversion warnings (#1172) 
* Fix sign conversion warnings

* Fix type conversion warnings

* Fix sign conversion warnings

* Change smem_size_ to constexpr

* clang warnings

* undo cast change

* one miss change

* missing part

---------

Co-authored-by: Haicheng Wu <haichengw@nvidia.com>
 2023-11-30 00:28:40 -05:00
Christian Sigg 99c4eebe3b Explicitly cast blockIdx to uint3 (#1192) 
This works around a clang issue where blockIdx is of a different type.
 2023-11-30 00:26:23 -05:00
Sergey KlevtsovandSergey Klevtsov b5d8a5d9cc Allow SM90 pingpong kernel to use custom tile schedulers (#1194) 
Co-authored-by: Sergey Klevtsov <sklevtsov@nvidia.com>
 2023-11-15 13:45:17 -05:00
wang-y-zandisaacw 557be3ab0e Fix several typos (#1169) 
Co-authored-by: isaacw <isaacw@nvidia.com>
 2023-11-02 23:54:46 -04:00
Pradeep Ramani c008b4aea8 CUTLASS 3.3.0 (#1167) 
* Release 3.3.0

Adds support for mixed precision GEMMs On Hopper and Ampere
Adds support for < 16B aligned GEMMs on Hopper
Enhancements to EVT
Enhancements to Python interface
Enhancements to Sub-byte type handling in CuTe
Several other bug-fixes and performance improvements.

* minor doc update
 2023-11-02 11:09:05 -04:00
Manish Gupta 7d8317a63e Support for Mixed Input TensorOp (#1084) 
* Passing warp-level mixed input F16*(S8/U8) tests

* passing device-level mixed input F16*(S8/U8) tests

* add to profiler - I8 (111 TFLOPs), U (123 TFLOPs)

* fast numeric conversions (I8 = 132 TFLOPs, U8 = 148 TFLOPs)

* Speedup reference compilation (REVERT THIS COMMIT)

* wider_add.u32_packed_sub.f16x2 (I8 = 132TFLOP/s, U8 = 170 TFLOP/s)

* Improve s8->f16 cvt and support bf16*u8 @158 TFLOPs

* BF16 * S8 (142 TFLOPs)

* Handle mixed-input upcast on OperandA (Support [S8|U8]*[F16|BF16]

* rename OpMultiplyAddMixedInput to OpMultiplyAddMixedInputUpcast

* Add device-level test and profiler support for upcast on operand A

* Move shfl before the cvt and reduce #shfls by 1/2

* fix smem_usage calculation for mixed_input types

* uncomment the stuff (getting ready for merge)

* profiler changes and mixed-input reference

* mixed input reference are in a new file

* use platform instead of std

* comments and typo only

* Use CreateGemmOperator and delete CreateMixedInputGemmOperator

* copyright for new files

* rebase follow-up
 2023-09-27 11:18:30 -04:00
ANIKET SHIVAM 90d3b0fb18 CUTLASS 3.2.1 (#1113) 
* Updates for 3.2.1 release.

* Minor fix in gemm op profiler for raster order.

* Add scheduler mapping for raster order in the kernels.
 2023-09-26 17:24:26 -04:00
Driss Guessous 88c0d7c726 make only visible on device (#1071) 2023-09-07 13:00:46 -04:00
ANIKET SHIVAM a88c41cf8d Updates for 3.2 release (#1065) 2023-08-25 23:05:46 -04:00