• 94 Posts
  • 153 Comments
Joined 1 year ago
cake
Cake day: June 12th, 2023

help-circle












  • Btw I know this is old and you may have already figured out your hardware and setup, but p40s and p100s go for super cheap on eBay.

    P40 is an amazing $/GB deal, only issue is the fp16 performance is abysmal so you’ll want to run either full fp32 models or use llama.cpp which is able to cast up to that size

    The p100 has less VRAM but really good fp16 performance which makes it ideal for exllamav2 usage. I picked up one of each recently, p40 was failed to deliver and p100 was delivered while I’m away, but once I have both on hand I’ll probably post a comparison to my 3090 for interests sake

    Also I run all my stuff on Linux (Ubuntu 22.04) with no issues




  • I don’t have a lot of experience with either at this time, I’ve used them here and there for programming questions but usually I stick to 7b models because I use them for code completion and I only find that useful if it completes the code before I do lol

    That said, I’ve had overall good answers from either whenever I’ve decided to pull them out, it feels like wizard coder should be better since it’s so much newer but overall it hasn’t been that different. Wish phind would release an update :(